-- Logs begin at Tue 2019-06-18 12:09:07 PDT, end at Mon 2019-07-15 15:39:01 PDT. -- Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys cpuset Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys cpu Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys cpuacct Jun 18 12:09:07 fir-md1-s1 kernel: Linux version 3.10.0-957.1.3.el7_lustre.x86_64 (sthiell@fir-io1-s1) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 18 12:09:07 fir-md1-s1 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.1.3.el7_lustre.x86_64 root=UUID=4adf0488-f60f-46c3-a712-956aaee5c4b2 ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 Jun 18 12:09:07 fir-md1-s1 kernel: e820: BIOS-provided physical RAM map: Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000008efff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000090000-0x000000000009ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000100000-0x000000005c3dffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000005c3e0000-0x00000000643e7fff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x00000000643e8000-0x000000006cacefff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006cacf000-0x000000006efcefff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x00000000fed80000-0x00000000fed80fff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000107f37ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000107f380000-0x000000107fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000001080000000-0x000000207ff7ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000207ff80000-0x000000207fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000002080000000-0x000000307ff7ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000307ff80000-0x000000307fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000003080000000-0x000000407ff7ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000407ff80000-0x000000407fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: NX (Execute Disable) protection: active Jun 18 12:09:07 fir-md1-s1 kernel: e820: update [mem 0x446da020-0x4470b25f] usable ==> usable Jun 18 12:09:07 fir-md1-s1 kernel: e820: update [mem 0x446a8020-0x446d925f] usable ==> usable Jun 18 12:09:07 fir-md1-s1 kernel: e820: update [mem 0x5b485020-0x5b48d05f] usable ==> usable Jun 18 12:09:07 fir-md1-s1 kernel: e820: update [mem 0x4468f020-0x446a765f] usable ==> usable Jun 18 12:09:07 fir-md1-s1 kernel: extended physical RAM map: Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000000000-0x000000000008efff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000090000-0x000000000009ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000100000-0x000000004468f01f] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000004468f020-0x00000000446a765f] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000446a7660-0x00000000446a801f] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000446a8020-0x00000000446d925f] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000446d9260-0x00000000446da01f] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000446da020-0x000000004470b25f] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000004470b260-0x000000005b48501f] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000005b485020-0x000000005b48d05f] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000005b48d060-0x000000005c3dffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000005c3e0000-0x00000000643e7fff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000643e8000-0x000000006cacefff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006cacf000-0x000000006efcefff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006ffff000-0x000000006fffffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000070000000-0x000000008fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000fec10000-0x00000000fec10fff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000fed80000-0x00000000fed80fff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000100000000-0x000000107f37ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000107f380000-0x000000107fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000001080000000-0x000000207ff7ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000207ff80000-0x000000207fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000002080000000-0x000000307ff7ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000307ff80000-0x000000307fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000003080000000-0x000000407ff7ffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000407ff80000-0x000000407fffffff] reserved Jun 18 12:09:07 fir-md1-s1 kernel: efi: EFI v2.50 by Dell Inc. Jun 18 12:09:07 fir-md1-s1 kernel: efi: ACPI=0x6fffe000 ACPI 2.0=0x6fffe014 SMBIOS=0x6eab5000 SMBIOS 3.0=0x6eab3000 Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem00: type=3, attr=0xf, range=[0x0000000000000000-0x0000000000001000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem01: type=2, attr=0xf, range=[0x0000000000001000-0x0000000000002000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem02: type=7, attr=0xf, range=[0x0000000000002000-0x0000000000010000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem03: type=3, attr=0xf, range=[0x0000000000010000-0x0000000000014000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem04: type=7, attr=0xf, range=[0x0000000000014000-0x0000000000063000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem05: type=3, attr=0xf, range=[0x0000000000063000-0x000000000008f000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem06: type=10, attr=0xf, range=[0x000000000008f000-0x0000000000090000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem07: type=3, attr=0xf, range=[0x0000000000090000-0x00000000000a0000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem08: type=4, attr=0xf, range=[0x0000000000100000-0x0000000000120000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem09: type=7, attr=0xf, range=[0x0000000000120000-0x0000000000c00000) (10MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem10: type=3, attr=0xf, range=[0x0000000000c00000-0x0000000001000000) (4MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem11: type=2, attr=0xf, range=[0x0000000001000000-0x000000000267a000) (22MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem12: type=7, attr=0xf, range=[0x000000000267a000-0x0000000004000000) (25MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem13: type=4, attr=0xf, range=[0x0000000004000000-0x000000000403b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem14: type=7, attr=0xf, range=[0x000000000403b000-0x000000003ecab000) (940MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem15: type=2, attr=0xf, range=[0x000000003ecab000-0x0000000040000000) (19MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem16: type=7, attr=0xf, range=[0x0000000040000000-0x000000004468f000) (70MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem17: type=2, attr=0xf, range=[0x000000004468f000-0x000000005b25c000) (363MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem18: type=1, attr=0xf, range=[0x000000005b25c000-0x000000005b475000) (2MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem19: type=7, attr=0xf, range=[0x000000005b475000-0x000000005b485000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem20: type=2, attr=0xf, range=[0x000000005b485000-0x000000005b48e000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem21: type=4, attr=0xf, range=[0x000000005b48e000-0x000000005b491000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem22: type=2, attr=0xf, range=[0x000000005b491000-0x000000005b59c000) (1MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem23: type=4, attr=0xf, range=[0x000000005b59c000-0x000000005b6bf000) (1MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem24: type=3, attr=0xf, range=[0x000000005b6bf000-0x000000005b70d000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem25: type=4, attr=0xf, range=[0x000000005b70d000-0x000000005b75b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem26: type=3, attr=0xf, range=[0x000000005b75b000-0x000000005b7bd000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem27: type=4, attr=0xf, range=[0x000000005b7bd000-0x000000005b7c7000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem28: type=3, attr=0xf, range=[0x000000005b7c7000-0x000000005b8b2000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem29: type=4, attr=0xf, range=[0x000000005b8b2000-0x000000005b8c1000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem30: type=7, attr=0xf, range=[0x000000005b8c1000-0x000000005b8c7000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem31: type=4, attr=0xf, range=[0x000000005b8c7000-0x000000005b8cc000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem32: type=3, attr=0xf, range=[0x000000005b8cc000-0x000000005b927000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem33: type=4, attr=0xf, range=[0x000000005b927000-0x000000005b931000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem34: type=3, attr=0xf, range=[0x000000005b931000-0x000000005b960000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem35: type=4, attr=0xf, range=[0x000000005b960000-0x000000005bc2f000) (2MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem36: type=3, attr=0xf, range=[0x000000005bc2f000-0x000000005bc3c000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem37: type=4, attr=0xf, range=[0x000000005bc3c000-0x000000005be3a000) (1MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem38: type=7, attr=0xf, range=[0x000000005be3a000-0x000000005be3b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem39: type=4, attr=0xf, range=[0x000000005be3b000-0x000000005be4f000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem40: type=7, attr=0xf, range=[0x000000005be4f000-0x000000005be50000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem41: type=4, attr=0xf, range=[0x000000005be50000-0x000000005be5a000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem42: type=7, attr=0xf, range=[0x000000005be5a000-0x000000005be5b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem43: type=4, attr=0xf, range=[0x000000005be5b000-0x000000005be61000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem44: type=2, attr=0xf, range=[0x000000005be61000-0x000000005be63000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem45: type=4, attr=0xf, range=[0x000000005be63000-0x000000005be64000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem46: type=7, attr=0xf, range=[0x000000005be64000-0x000000005be65000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem47: type=4, attr=0xf, range=[0x000000005be65000-0x000000005be77000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem48: type=7, attr=0xf, range=[0x000000005be77000-0x000000005be78000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem49: type=4, attr=0xf, range=[0x000000005be78000-0x000000005be79000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem50: type=7, attr=0xf, range=[0x000000005be79000-0x000000005be7a000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem51: type=2, attr=0xf, range=[0x000000005be7a000-0x000000005be7b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem52: type=4, attr=0xf, range=[0x000000005be7b000-0x000000005be7f000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem53: type=3, attr=0xf, range=[0x000000005be7f000-0x000000005bea2000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem54: type=4, attr=0xf, range=[0x000000005bea2000-0x000000005c003000) (1MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem55: type=3, attr=0xf, range=[0x000000005c003000-0x000000005c3e0000) (3MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem56: type=0, attr=0xf, range=[0x000000005c3e0000-0x00000000643e8000) (128MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem57: type=3, attr=0xf, range=[0x00000000643e8000-0x0000000064fae000) (11MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem58: type=4, attr=0xf, range=[0x0000000064fae000-0x0000000068acf000) (59MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem59: type=3, attr=0xf, range=[0x0000000068acf000-0x0000000068ecf000) (4MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem60: type=7, attr=0xf, range=[0x0000000068ecf000-0x0000000068ed1000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem61: type=4, attr=0xf, range=[0x0000000068ed1000-0x0000000069066000) (1MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem62: type=7, attr=0xf, range=[0x0000000069066000-0x0000000069067000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem63: type=4, attr=0xf, range=[0x0000000069067000-0x000000006907a000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem64: type=7, attr=0xf, range=[0x000000006907a000-0x000000006907b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem65: type=4, attr=0xf, range=[0x000000006907b000-0x0000000069087000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem66: type=7, attr=0xf, range=[0x0000000069087000-0x0000000069088000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem67: type=4, attr=0xf, range=[0x0000000069088000-0x0000000069089000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem68: type=7, attr=0xf, range=[0x0000000069089000-0x000000006908a000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem69: type=4, attr=0xf, range=[0x000000006908a000-0x0000000069097000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem70: type=7, attr=0xf, range=[0x0000000069097000-0x0000000069098000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem71: type=4, attr=0xf, range=[0x0000000069098000-0x000000006909b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem72: type=7, attr=0xf, range=[0x000000006909b000-0x000000006909c000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem73: type=4, attr=0xf, range=[0x000000006909c000-0x00000000690dc000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem74: type=7, attr=0xf, range=[0x00000000690dc000-0x00000000690dd000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem75: type=4, attr=0xf, range=[0x00000000690dd000-0x000000006911a000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem76: type=7, attr=0xf, range=[0x000000006911a000-0x000000006911b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem77: type=4, attr=0xf, range=[0x000000006911b000-0x000000006911f000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem78: type=7, attr=0xf, range=[0x000000006911f000-0x0000000069120000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem79: type=4, attr=0xf, range=[0x0000000069120000-0x000000006914c000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem80: type=7, attr=0xf, range=[0x000000006914c000-0x000000006914d000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem81: type=4, attr=0xf, range=[0x000000006914d000-0x0000000069152000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem82: type=7, attr=0xf, range=[0x0000000069152000-0x0000000069153000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem83: type=4, attr=0xf, range=[0x0000000069153000-0x0000000069164000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem84: type=7, attr=0xf, range=[0x0000000069164000-0x0000000069165000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem85: type=4, attr=0xf, range=[0x0000000069165000-0x000000006917a000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem86: type=7, attr=0xf, range=[0x000000006917a000-0x000000006917b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem87: type=4, attr=0xf, range=[0x000000006917b000-0x000000006918b000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem88: type=7, attr=0xf, range=[0x000000006918b000-0x000000006918c000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem89: type=4, attr=0xf, range=[0x000000006918c000-0x00000000691ea000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem90: type=7, attr=0xf, range=[0x00000000691ea000-0x00000000691eb000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem91: type=4, attr=0xf, range=[0x00000000691eb000-0x00000000691ff000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem92: type=7, attr=0xf, range=[0x00000000691ff000-0x0000000069200000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem93: type=4, attr=0xf, range=[0x0000000069200000-0x0000000069204000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem94: type=7, attr=0xf, range=[0x0000000069204000-0x0000000069205000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem95: type=4, attr=0xf, range=[0x0000000069205000-0x000000006920e000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem96: type=7, attr=0xf, range=[0x000000006920e000-0x000000006920f000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem97: type=4, attr=0xf, range=[0x000000006920f000-0x0000000069216000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem98: type=7, attr=0xf, range=[0x0000000069216000-0x0000000069217000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem99: type=4, attr=0xf, range=[0x0000000069217000-0x0000000069218000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem100: type=7, attr=0xf, range=[0x0000000069218000-0x0000000069219000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem101: type=4, attr=0xf, range=[0x0000000069219000-0x000000006921c000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem102: type=7, attr=0xf, range=[0x000000006921c000-0x000000006921e000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem103: type=4, attr=0xf, range=[0x000000006921e000-0x0000000069223000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem104: type=7, attr=0xf, range=[0x0000000069223000-0x0000000069224000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem105: type=4, attr=0xf, range=[0x0000000069224000-0x0000000069226000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem106: type=7, attr=0xf, range=[0x0000000069226000-0x0000000069227000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem107: type=4, attr=0xf, range=[0x0000000069227000-0x000000006922f000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem108: type=7, attr=0xf, range=[0x000000006922f000-0x0000000069230000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem109: type=4, attr=0xf, range=[0x0000000069230000-0x000000006924f000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem110: type=7, attr=0xf, range=[0x000000006924f000-0x0000000069250000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem111: type=4, attr=0xf, range=[0x0000000069250000-0x000000006a2d3000) (16MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem112: type=7, attr=0xf, range=[0x000000006a2d3000-0x000000006a2d5000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem113: type=4, attr=0xf, range=[0x000000006a2d5000-0x000000006c3cf000) (32MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem114: type=7, attr=0xf, range=[0x000000006c3cf000-0x000000006c3d1000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem115: type=3, attr=0xf, range=[0x000000006c3d1000-0x000000006cacf000) (6MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem116: type=6, attr=0x800000000000000f, range=[0x000000006cacf000-0x000000006cbcf000) (1MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem117: type=5, attr=0x800000000000000f, range=[0x000000006cbcf000-0x000000006cdcf000) (2MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem118: type=0, attr=0xf, range=[0x000000006cdcf000-0x000000006efcf000) (34MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem119: type=10, attr=0xf, range=[0x000000006efcf000-0x000000006fdff000) (14MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem120: type=9, attr=0xf, range=[0x000000006fdff000-0x000000006ffff000) (2MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem121: type=4, attr=0xf, range=[0x000000006ffff000-0x0000000070000000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem122: type=7, attr=0xf, range=[0x0000000100000000-0x000000107f380000) (63475MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem123: type=7, attr=0xf, range=[0x0000001080000000-0x000000207ff80000) (65535MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem124: type=7, attr=0xf, range=[0x0000002080000000-0x000000307ff80000) (65535MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem125: type=7, attr=0xf, range=[0x0000003080000000-0x000000407ff80000) (65535MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem126: type=0, attr=0x9, range=[0x0000000070000000-0x0000000080000000) (256MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem127: type=11, attr=0x800000000000000f, range=[0x0000000080000000-0x0000000090000000) (256MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem128: type=11, attr=0x800000000000000f, range=[0x00000000fec10000-0x00000000fec11000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem129: type=11, attr=0x800000000000000f, range=[0x00000000fed80000-0x00000000fed81000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem130: type=0, attr=0x0, range=[0x000000107f380000-0x0000001080000000) (12MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem131: type=0, attr=0x0, range=[0x000000207ff80000-0x0000002080000000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem132: type=0, attr=0x0, range=[0x000000307ff80000-0x0000003080000000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: efi: mem133: type=0, attr=0x0, range=[0x000000407ff80000-0x0000004080000000) (0MB) Jun 18 12:09:07 fir-md1-s1 kernel: SMBIOS 3.0.0 present. Jun 18 12:09:07 fir-md1-s1 kernel: DMI: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Jun 18 12:09:07 fir-md1-s1 kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved Jun 18 12:09:07 fir-md1-s1 kernel: e820: remove [mem 0x000a0000-0x000fffff] usable Jun 18 12:09:07 fir-md1-s1 kernel: e820: last_pfn = 0x407ff80 max_arch_pfn = 0x400000000 Jun 18 12:09:07 fir-md1-s1 kernel: MTRR default type: uncachable Jun 18 12:09:07 fir-md1-s1 kernel: MTRR fixed ranges enabled: Jun 18 12:09:07 fir-md1-s1 kernel: 00000-9FFFF write-back Jun 18 12:09:07 fir-md1-s1 kernel: A0000-FFFFF uncachable Jun 18 12:09:07 fir-md1-s1 kernel: MTRR variable ranges enabled: Jun 18 12:09:07 fir-md1-s1 kernel: 0 base 0000FF000000 mask FFFFFF000000 write-protect Jun 18 12:09:07 fir-md1-s1 kernel: 1 base 000000000000 mask FFFF80000000 write-back Jun 18 12:09:07 fir-md1-s1 kernel: 2 base 000070000000 mask FFFFF0000000 uncachable Jun 18 12:09:07 fir-md1-s1 kernel: 3 disabled Jun 18 12:09:07 fir-md1-s1 kernel: 4 disabled Jun 18 12:09:07 fir-md1-s1 kernel: 5 disabled Jun 18 12:09:07 fir-md1-s1 kernel: 6 disabled Jun 18 12:09:07 fir-md1-s1 kernel: 7 disabled Jun 18 12:09:07 fir-md1-s1 kernel: TOM2: 0000004080000000 aka 264192M Jun 18 12:09:07 fir-md1-s1 kernel: PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC Jun 18 12:09:07 fir-md1-s1 kernel: e820: last_pfn = 0x70000 max_arch_pfn = 0x400000000 Jun 18 12:09:07 fir-md1-s1 kernel: Base memory trampoline at [ffff8f0500099000] 99000 size 24576 Jun 18 12:09:07 fir-md1-s1 kernel: Using GB pages for direct mapping Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc52000, 0x2b0fc52fff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc53000, 0x2b0fc53fff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc54000, 0x2b0fc54fff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc55000, 0x2b0fc55fff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc56000, 0x2b0fc56fff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc57000, 0x2b0fc57fff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc58000, 0x2b0fc58fff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc59000, 0x2b0fc59fff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc5a000, 0x2b0fc5afff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc5b000, 0x2b0fc5bfff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc5c000, 0x2b0fc5cfff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: BRK [0x2b0fc5d000, 0x2b0fc5dfff] PGTABLE Jun 18 12:09:07 fir-md1-s1 kernel: RAMDISK: [mem 0x3ecab000-0x3fffdfff] Jun 18 12:09:07 fir-md1-s1 kernel: Early table checksum verification disabled Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: RSDP 000000006fffe014 00024 (v02 DELL ) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: XSDT 000000006fffd0e8 000B4 (v01 DELL PE_SC3 00000002 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: FACP 000000006ffef000 00114 (v06 DELL PE_SC3 00000002 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: DSDT 000000006ffe2000 0950B (v02 DELL PE_SC3 00000002 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: FACS 000000006fdd4000 00040 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: SSDT 000000006fffc000 000D2 (v02 DELL PE_SC3 00000002 MSFT 04000000) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: BERT 000000006fffb000 00030 (v01 DELL BERT 00000001 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: HEST 000000006fffa000 006DC (v01 DELL HEST 00000001 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: SSDT 000000006fff9000 00294 (v01 DELL PE_SC3 00000001 AMD 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: SRAT 000000006fff8000 00420 (v03 DELL PE_SC3 00000001 AMD 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: MSCT 000000006fff7000 0004E (v01 DELL PE_SC3 00000000 AMD 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: SLIT 000000006fff6000 0003C (v01 DELL PE_SC3 00000001 AMD 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: CRAT 000000006fff3000 02DC0 (v01 DELL PE_SC3 00000001 AMD 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: CDIT 000000006fff2000 00038 (v01 DELL PE_SC3 00000001 AMD 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: EINJ 000000006fff1000 00150 (v01 DELL PE_SC3 00000001 AMD 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: SLIC 000000006fff0000 00024 (v01 DELL PE_SC3 00000002 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: HPET 000000006ffee000 00038 (v01 DELL PE_SC3 00000002 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: APIC 000000006ffed000 004B2 (v03 DELL PE_SC3 00000002 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: MCFG 000000006ffec000 0003C (v01 DELL PE_SC3 00000002 DELL 00000001) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: SSDT 000000006ffe1000 00629 (v02 DELL xhc_port 00000001 INTL 20170119) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: IVRS 000000006ffe0000 00210 (v02 DELL PE_SC3 00000001 AMD 00000000) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: SSDT 000000006ffde000 01658 (v01 AMD CPMCMN 00000001 INTL 20170119) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Local APIC address 0xfee00000 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x00 -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x01 -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x02 -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x03 -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x04 -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x05 -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x08 -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x09 -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0a -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0b -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0c -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0d -> Node 0 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x10 -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x11 -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x12 -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x13 -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x14 -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x15 -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x18 -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x19 -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1a -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1b -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1c -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1d -> Node 1 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x20 -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x21 -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x22 -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x23 -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x24 -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x25 -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x28 -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x29 -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2a -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2b -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2c -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2d -> Node 2 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x30 -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x31 -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x32 -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x33 -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x34 -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x35 -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x38 -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x39 -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3a -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3b -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3c -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3d -> Node 3 Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x00100000-0x7fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: Node 2 PXM 2 [mem 0x2080000000-0x307fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: SRAT: Node 3 PXM 3 [mem 0x3080000000-0x407fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: NUMA: Initialized distance table, cnt=4 Jun 18 12:09:07 fir-md1-s1 kernel: NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x7fffffff] -> [mem 0x00000000-0x7fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [mem 0x00000000-0x107fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: NODE_DATA(0) allocated [mem 0x107f359000-0x107f37ffff] Jun 18 12:09:07 fir-md1-s1 kernel: NODE_DATA(1) allocated [mem 0x207ff59000-0x207ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: NODE_DATA(2) allocated [mem 0x307ff59000-0x307ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: NODE_DATA(3) allocated [mem 0x407ff58000-0x407ff7efff] Jun 18 12:09:07 fir-md1-s1 kernel: Reserving 176MB of memory at 720MB for crashkernel (System RAM: 261692MB) Jun 18 12:09:07 fir-md1-s1 kernel: Zone ranges: Jun 18 12:09:07 fir-md1-s1 kernel: DMA [mem 0x00001000-0x00ffffff] Jun 18 12:09:07 fir-md1-s1 kernel: DMA32 [mem 0x01000000-0xffffffff] Jun 18 12:09:07 fir-md1-s1 kernel: Normal [mem 0x100000000-0x407ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: Movable zone start for each node Jun 18 12:09:07 fir-md1-s1 kernel: Early memory node ranges Jun 18 12:09:07 fir-md1-s1 kernel: node 0: [mem 0x00001000-0x0008efff] Jun 18 12:09:07 fir-md1-s1 kernel: node 0: [mem 0x00090000-0x0009ffff] Jun 18 12:09:07 fir-md1-s1 kernel: node 0: [mem 0x00100000-0x5c3dffff] Jun 18 12:09:07 fir-md1-s1 kernel: node 0: [mem 0x643e8000-0x6cacefff] Jun 18 12:09:07 fir-md1-s1 kernel: node 0: [mem 0x6ffff000-0x6fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: node 0: [mem 0x100000000-0x107f37ffff] Jun 18 12:09:07 fir-md1-s1 kernel: node 1: [mem 0x1080000000-0x207ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: node 2: [mem 0x2080000000-0x307ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: node 3: [mem 0x3080000000-0x407ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: Initmem setup node 0 [mem 0x00001000-0x107f37ffff] Jun 18 12:09:07 fir-md1-s1 kernel: On node 0 totalpages: 16661990 Jun 18 12:09:07 fir-md1-s1 kernel: DMA zone: 64 pages used for memmap Jun 18 12:09:07 fir-md1-s1 kernel: DMA zone: 1126 pages reserved Jun 18 12:09:07 fir-md1-s1 kernel: DMA zone: 3998 pages, LIFO batch:0 Jun 18 12:09:07 fir-md1-s1 kernel: DMA32 zone: 6380 pages used for memmap Jun 18 12:09:07 fir-md1-s1 kernel: DMA32 zone: 408264 pages, LIFO batch:31 Jun 18 12:09:07 fir-md1-s1 kernel: Normal zone: 253902 pages used for memmap Jun 18 12:09:07 fir-md1-s1 kernel: Normal zone: 16249728 pages, LIFO batch:31 Jun 18 12:09:07 fir-md1-s1 kernel: Initmem setup node 1 [mem 0x1080000000-0x207ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: On node 1 totalpages: 16777088 Jun 18 12:09:07 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Jun 18 12:09:07 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Jun 18 12:09:07 fir-md1-s1 kernel: Initmem setup node 2 [mem 0x2080000000-0x307ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: On node 2 totalpages: 16777088 Jun 18 12:09:07 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Jun 18 12:09:07 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Jun 18 12:09:07 fir-md1-s1 kernel: Initmem setup node 3 [mem 0x3080000000-0x407ff7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: On node 3 totalpages: 16777088 Jun 18 12:09:07 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Jun 18 12:09:07 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PM-Timer IO Port: 0x408 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Local APIC address 0xfee00000 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x20] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x30] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x18] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x28] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x38] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x02] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x12] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x22] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0a] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x1a] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2a] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x3a] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x04] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x14] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x24] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x34] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x0c] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x1c] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x2c] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x3c] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x01] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0x11] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x21] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x31] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x09] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x19] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x29] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x39] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0x03] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0x13] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0x23] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0x33] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0x0b] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0x1b] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0x2b] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0x3b] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0x05] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0x15] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x25] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x35] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x0d] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x1d] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x2d] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x3d] enabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x40] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x41] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x42] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x43] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x44] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x45] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x46] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x47] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x48] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x49] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4a] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4b] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4c] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4d] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4e] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4f] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x50] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x51] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x52] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x53] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x54] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x55] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x56] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x57] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x58] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x59] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5a] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5b] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5c] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5d] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5e] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5f] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x60] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x61] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x62] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x63] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x64] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x65] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x66] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x67] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x68] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x69] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6a] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6b] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6c] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6d] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6e] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6f] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x70] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x71] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x72] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x73] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x74] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x75] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x76] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x77] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x78] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x79] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7a] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7b] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7c] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7d] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7e] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7f] lapic_id[0x00] disabled) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x80] address[0xfec00000] gsi_base[0]) Jun 18 12:09:07 fir-md1-s1 kernel: IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x81] address[0xfd880000] gsi_base[24]) Jun 18 12:09:07 fir-md1-s1 kernel: IOAPIC[1]: apic_id 129, version 33, address 0xfd880000, GSI 24-55 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x82] address[0xe0900000] gsi_base[56]) Jun 18 12:09:07 fir-md1-s1 kernel: IOAPIC[2]: apic_id 130, version 33, address 0xe0900000, GSI 56-87 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x83] address[0xc5900000] gsi_base[88]) Jun 18 12:09:07 fir-md1-s1 kernel: IOAPIC[3]: apic_id 131, version 33, address 0xc5900000, GSI 88-119 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x84] address[0xaa900000] gsi_base[120]) Jun 18 12:09:07 fir-md1-s1 kernel: IOAPIC[4]: apic_id 132, version 33, address 0xaa900000, GSI 120-151 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: IRQ0 used by override. Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: IRQ9 used by override. Jun 18 12:09:07 fir-md1-s1 kernel: Using ACPI (MADT) for SMP configuration information Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: HPET id: 0x10228201 base: 0xfed00000 Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Allowing 128 CPUs, 80 hotplug CPUs Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x0008f000-0x0008ffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x000a0000-0x000fffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x4468f000-0x4468ffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x446a7000-0x446a7fff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x446a8000-0x446a8fff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x446d9000-0x446d9fff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x446da000-0x446dafff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x4470b000-0x4470bfff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x5b485000-0x5b485fff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x5b48d000-0x5b48dfff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x5c3e0000-0x643e7fff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6cacf000-0x6efcefff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6efcf000-0x6fdfefff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6fdff000-0x6fffefff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x70000000-0x8fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x90000000-0xfec0ffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfec10000-0xfec10fff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfec11000-0xfed7ffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfed80000-0xfed80fff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfed81000-0xffffffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x107f380000-0x107fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x207ff80000-0x207fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x307ff80000-0x307fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: [mem 0x90000000-0xfec0ffff] available for PCI devices Jun 18 12:09:07 fir-md1-s1 kernel: Booting paravirtualized kernel on bare hardware Jun 18 12:09:07 fir-md1-s1 kernel: setup_percpu: NR_CPUS:5120 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:4 Jun 18 12:09:07 fir-md1-s1 kernel: PERCPU: Embedded 38 pages/cpu @ffff8f153ee00000 s118784 r8192 d28672 u262144 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [0] 000 004 008 012 016 020 024 028 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [0] 032 036 040 044 048 052 056 060 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [0] 064 068 072 076 080 084 088 092 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [0] 096 100 104 108 112 116 120 124 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [1] 001 005 009 013 017 021 025 029 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [1] 033 037 041 045 049 053 057 061 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [1] 065 069 073 077 081 085 089 093 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [1] 097 101 105 109 113 117 121 125 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [2] 002 006 010 014 018 022 026 030 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [2] 034 038 042 046 050 054 058 062 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [2] 066 070 074 078 082 086 090 094 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [2] 098 102 106 110 114 118 122 126 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [3] 003 007 011 015 019 023 027 031 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [3] 035 039 043 047 051 055 059 063 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [3] 067 071 075 079 083 087 091 095 Jun 18 12:09:07 fir-md1-s1 kernel: pcpu-alloc: [3] 099 103 107 111 115 119 123 127 Jun 18 12:09:07 fir-md1-s1 kernel: Built 4 zonelists in Zone order, mobility grouping on. Total pages: 65945356 Jun 18 12:09:07 fir-md1-s1 kernel: Policy zone: Normal Jun 18 12:09:07 fir-md1-s1 kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.1.3.el7_lustre.x86_64 root=UUID=4adf0488-f60f-46c3-a712-956aaee5c4b2 ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 Jun 18 12:09:07 fir-md1-s1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 Jun 18 12:09:07 fir-md1-s1 kernel: xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form Jun 18 12:09:07 fir-md1-s1 kernel: Memory: 9614216k/270532096k available (7664k kernel code, 2559080k absent, 4653740k reserved, 6055k data, 1876k init) Jun 18 12:09:07 fir-md1-s1 kernel: SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=4 Jun 18 12:09:07 fir-md1-s1 kernel: Hierarchical RCU implementation. Jun 18 12:09:07 fir-md1-s1 kernel: RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=128. Jun 18 12:09:07 fir-md1-s1 kernel: NR_IRQS:327936 nr_irqs:3624 0 Jun 18 12:09:07 fir-md1-s1 kernel: Console: colour dummy device 80x25 Jun 18 12:09:07 fir-md1-s1 kernel: console [ttyS0] enabled Jun 18 12:09:07 fir-md1-s1 kernel: allocated 1072693248 bytes of page_cgroup Jun 18 12:09:07 fir-md1-s1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Jun 18 12:09:07 fir-md1-s1 kernel: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl Jun 18 12:09:07 fir-md1-s1 kernel: hpet clockevent registered Jun 18 12:09:07 fir-md1-s1 kernel: tsc: Fast TSC calibration using PIT Jun 18 12:09:07 fir-md1-s1 kernel: tsc: Detected 1996.233 MHz processor Jun 18 12:09:07 fir-md1-s1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3992.46 BogoMIPS (lpj=1996233) Jun 18 12:09:07 fir-md1-s1 kernel: pid_max: default: 131072 minimum: 1024 Jun 18 12:09:07 fir-md1-s1 kernel: Security Framework initialized Jun 18 12:09:07 fir-md1-s1 kernel: SELinux: Initializing. Jun 18 12:09:07 fir-md1-s1 kernel: SELinux: Starting in permissive mode Jun 18 12:09:07 fir-md1-s1 kernel: Yama: becoming mindful. Jun 18 12:09:07 fir-md1-s1 kernel: Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: random: fast init done Jun 18 12:09:07 fir-md1-s1 kernel: Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys memory Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys devices Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys freezer Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys net_cls Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys blkio Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys perf_event Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys hugetlb Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys pids Jun 18 12:09:07 fir-md1-s1 kernel: Initializing cgroup subsys net_prio Jun 18 12:09:07 fir-md1-s1 kernel: tseg: 0070000000 Jun 18 12:09:07 fir-md1-s1 kernel: mce: CPU supports 23 MCE banks Jun 18 12:09:07 fir-md1-s1 kernel: LVT offset 2 assigned for vector 0xf4 Jun 18 12:09:07 fir-md1-s1 kernel: Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 512 Jun 18 12:09:07 fir-md1-s1 kernel: Last level dTLB entries: 4KB 1536, 2MB 1536, 4MB 768 Jun 18 12:09:07 fir-md1-s1 kernel: tlb_flushall_shift: 6 Jun 18 12:09:07 fir-md1-s1 kernel: Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp Jun 18 12:09:07 fir-md1-s1 kernel: FEATURE SPEC_CTRL Not Present Jun 18 12:09:07 fir-md1-s1 kernel: FEATURE IBPB_SUPPORT Present Jun 18 12:09:07 fir-md1-s1 kernel: Spectre V2 : Mitigation: Full retpoline Jun 18 12:09:07 fir-md1-s1 kernel: Freeing SMP alternatives: 28k freed Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Core revision 20130517 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: All ACPI Tables successfully acquired Jun 18 12:09:07 fir-md1-s1 kernel: ftrace: allocating 29188 entries in 115 pages Jun 18 12:09:07 fir-md1-s1 kernel: Switched APIC routing to physical flat. Jun 18 12:09:07 fir-md1-s1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: CPU0: AMD EPYC 7401P 24-Core Processor (fam: 17, model: 01, stepping: 02) Jun 18 12:09:07 fir-md1-s1 kernel: Performance Events: Fam17h core perfctr, AMD PMU driver. Jun 18 12:09:07 fir-md1-s1 kernel: ... version: 0 Jun 18 12:09:07 fir-md1-s1 kernel: ... bit width: 48 Jun 18 12:09:07 fir-md1-s1 kernel: ... generic registers: 6 Jun 18 12:09:07 fir-md1-s1 kernel: ... value mask: 0000ffffffffffff Jun 18 12:09:07 fir-md1-s1 kernel: ... max period: 00007fffffffffff Jun 18 12:09:07 fir-md1-s1 kernel: ... fixed-purpose events: 0 Jun 18 12:09:07 fir-md1-s1 kernel: ... event mask: 000000000000003f Jun 18 12:09:07 fir-md1-s1 kernel: NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #1 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #2 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #3 OK Jun 18 12:09:07 fir-md1-s1 kernel: do_IRQ: 4.55 No irq handler for vector (irq -1) Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #4 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors Jun 18 12:09:07 fir-md1-s1 kernel: #5 Jun 18 12:09:07 fir-md1-s1 kernel: OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors Jun 18 12:09:07 fir-md1-s1 kernel: #6 Jun 18 12:09:07 fir-md1-s1 kernel: OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors Jun 18 12:09:07 fir-md1-s1 kernel: #7 Jun 18 12:09:07 fir-md1-s1 kernel: OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #8 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #9 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #10 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #11 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #12 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #13 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #14 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #15 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #16 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #17 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #18 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #19 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #20 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #21 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #22 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #23 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #24 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #25 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #26 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #27 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #28 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #29 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #30 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #31 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #32 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #33 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #34 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #35 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #36 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #37 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #38 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #39 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #40 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #41 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #42 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #43 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #44 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #45 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #46 OK Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #47 Jun 18 12:09:07 fir-md1-s1 kernel: Brought up 48 CPUs Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Max logical packages: 3 Jun 18 12:09:07 fir-md1-s1 kernel: smpboot: Total of 48 processors activated (191638.36 BogoMIPS) Jun 18 12:09:07 fir-md1-s1 kernel: node 0 initialised, 15462980 pages in 284ms Jun 18 12:09:07 fir-md1-s1 kernel: node 2 initialised, 15984665 pages in 289ms Jun 18 12:09:07 fir-md1-s1 kernel: node 3 initialised, 15989251 pages in 289ms Jun 18 12:09:07 fir-md1-s1 kernel: node 1 initialised, 15989367 pages in 289ms Jun 18 12:09:07 fir-md1-s1 kernel: devtmpfs: initialized Jun 18 12:09:07 fir-md1-s1 kernel: EVM: security.selinux Jun 18 12:09:07 fir-md1-s1 kernel: EVM: security.ima Jun 18 12:09:07 fir-md1-s1 kernel: EVM: security.capability Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registering ACPI NVS region [mem 0x0008f000-0x0008ffff] (4096 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: PM: Registering ACPI NVS region [mem 0x6efcf000-0x6fdfefff] (14876672 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: atomic64 test passed for x86-64 platform with CX8 and with SSE Jun 18 12:09:07 fir-md1-s1 kernel: pinctrl core: initialized pinctrl subsystem Jun 18 12:09:07 fir-md1-s1 kernel: RTC time: 19:09:02, date: 06/18/19 Jun 18 12:09:07 fir-md1-s1 kernel: NET: Registered protocol family 16 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: bus type PCI registered Jun 18 12:09:07 fir-md1-s1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Jun 18 12:09:07 fir-md1-s1 kernel: PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) Jun 18 12:09:07 fir-md1-s1 kernel: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 Jun 18 12:09:07 fir-md1-s1 kernel: PCI: Using configuration type 1 for base access Jun 18 12:09:07 fir-md1-s1 kernel: PCI: Dell System detected, enabling pci=bfsort. Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Added _OSI(Module Device) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Added _OSI(Processor Device) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Added _OSI(3.0 _SCP Extensions) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Added _OSI(Processor Aggregator Device) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Added _OSI(Linux-Dell-Video) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: EC: Look up EC in DSDT Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Executed 2 blocks of module-level executable AML code Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Interpreter enabled Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: (supports S0 S5) Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Using IOAPIC for interrupt routing Jun 18 12:09:07 fir-md1-s1 kernel: HEST: Table parsing has been initialized. Jun 18 12:09:07 fir-md1-s1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Enabled 1 GPEs in block 00 to 1F Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 4 5 7 10 11 14 15) *0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 4 5 7 10 11 14 15) *0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 4 5 7 10 11 14 15) *0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 4 5 7 10 11 14 15) *0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 4 5 7 10 11 14 15) *0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 7 10 11 14 15) *0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 4 5 7 10 11 14 15) *0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 7 10 11 14 15) *0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00-3f]) Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:00: PCIe AER handled by firmware Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration Jun 18 12:09:07 fir-md1-s1 kernel: PCI host bridge to bus 0000:00 Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x0000-0x03af window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x0d00-0x3fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0xe1000000-0xfebfffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x10000000000-0x2bf3fffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [bus 00-3f] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:00.0: [1022:1450] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:00.2: [1022:1451] type 00 class 0x080600 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:01.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:02.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: [1022:1453] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:04.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:07.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:07.1: [1022:1454] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:07.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:08.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:08.1: [1022:1454] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:08.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:14.0: [1022:790b] type 00 class 0x0c0500 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:14.3: [1022:790e] type 00 class 0x060100 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:18.0: [1022:1460] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:18.1: [1022:1461] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:18.2: [1022:1462] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:18.3: [1022:1463] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:18.4: [1022:1464] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:18.5: [1022:1465] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:18.6: [1022:1466] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:18.7: [1022:1467] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:19.0: [1022:1460] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:19.1: [1022:1461] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:19.2: [1022:1462] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:19.3: [1022:1463] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:19.4: [1022:1464] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:19.5: [1022:1465] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:19.6: [1022:1466] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:19.7: [1022:1467] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1a.0: [1022:1460] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1a.1: [1022:1461] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1a.2: [1022:1462] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1a.3: [1022:1463] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1a.4: [1022:1464] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1a.5: [1022:1465] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1a.6: [1022:1466] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1a.7: [1022:1467] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1b.0: [1022:1460] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1b.1: [1022:1461] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1b.2: [1022:1462] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1b.3: [1022:1463] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1b.4: [1022:1464] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1b.5: [1022:1465] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1b.6: [1022:1466] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:1b.7: [1022:1467] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: [1000:00d1] type 00 class 0x010700 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x10: [mem 0xe1000000-0xe10fffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x18: [mem 0xe1100000-0xe11fffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x20: [mem 0xf7500000-0xf75fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x24: [io 0x1000-0x10ff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: supports D1 D2 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: PCI bridge to [bus 01] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [io 0x1000-0x1fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xf7500000-0xf75fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xe1000000-0xe11fffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.0: [1022:145a] type 00 class 0x130000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.2: [1022:1456] type 00 class 0x108000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.2: reg 0x18: [mem 0xf7300000-0xf73fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.2: reg 0x24: [mem 0xf7400000-0xf7401fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.3: [1022:145f] type 00 class 0x0c0330 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.3: reg 0x10: [mem 0xf7200000-0xf72fffff 64bit] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:07.1: PCI bridge to [bus 02] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:03:00.0: [1022:1455] type 00 class 0x130000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:03:00.1: [1022:1468] type 00 class 0x108000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:03:00.1: reg 0x18: [mem 0xf7000000-0xf70fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:03:00.1: reg 0x24: [mem 0xf7100000-0xf7101fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:08.1: PCI bridge to [bus 03] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: on NUMA node 0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC01] (domain 0000 [bus 40-7f]) Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:01: PCIe AER handled by firmware Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration Jun 18 12:09:07 fir-md1-s1 kernel: PCI host bridge to bus 0000:40 Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [io 0x4000-0x7fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [mem 0xc6000000-0xe0ffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [mem 0x2bf40000000-0x47e7fffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [bus 40-7f] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:00.0: [1022:1450] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:00.2: [1022:1451] type 00 class 0x080600 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:01.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:02.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:03.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:04.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:07.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:07.1: [1022:1454] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:07.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:08.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:08.1: [1022:1454] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:08.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.0: [1022:145a] type 00 class 0x130000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.2: [1022:1456] type 00 class 0x108000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.2: reg 0x18: [mem 0xdb300000-0xdb3fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.2: reg 0x24: [mem 0xdb400000-0xdb401fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.3: [1022:145f] type 00 class 0x0c0330 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.3: reg 0x10: [mem 0xdb200000-0xdb2fffff 64bit] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.3: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:07.1: PCI bridge to [bus 41] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:42:00.0: [1022:1455] type 00 class 0x130000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:42:00.1: [1022:1468] type 00 class 0x108000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:42:00.1: reg 0x18: [mem 0xdb000000-0xdb0fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:42:00.1: reg 0x24: [mem 0xdb100000-0xdb101fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:08.1: PCI bridge to [bus 42] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:40: on NUMA node 1 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC02] (domain 0000 [bus 80-bf]) Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:02: PCIe AER handled by firmware Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: platform does not support [SHPCHotplug] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:02: FADT indicates ASPM is unsupported, using BIOS configuration Jun 18 12:09:07 fir-md1-s1 kernel: PCI host bridge to bus 0000:80 Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [io 0x03b0-0x03df window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [io 0x8000-0xbfff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0xab000000-0xc5ffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0x47e80000000-0x63dbfffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [bus 80-bf] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:00.0: [1022:1450] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:00.2: [1022:1451] type 00 class 0x080600 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.1: [1022:1453] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.2: [1022:1453] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.2: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:02.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.1: [1022:1453] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:04.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:07.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:07.1: [1022:1454] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:07.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:08.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:08.1: [1022:1454] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:08.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: [14e4:165f] type 00 class 0x020000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x10: [mem 0xaf030000-0xaf03ffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x18: [mem 0xaf040000-0xaf04ffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x20: [mem 0xaf050000-0xaf05ffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: [14e4:165f] type 00 class 0x020000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x10: [mem 0xaf000000-0xaf00ffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x18: [mem 0xaf010000-0xaf01ffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x20: [mem 0xaf020000-0xaf02ffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.1: PCI bridge to [bus 81] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xaf000000-0xaf0fffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:82:00.0: [1556:be00] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.2: PCI bridge to [bus 82-83] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xae000000-0xaeffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:83:00.0: [102b:0536] type 00 class 0x030000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x10: [mem 0xae000000-0xaeffffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x14: [mem 0xc0808000-0xc080bfff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x18: [mem 0xc0000000-0xc07fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:82:00.0: PCI bridge to [bus 83] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xae000000-0xaeffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:84:00.0: [15b3:1013] type 00 class 0x020700 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x10: [mem 0xac000000-0xadffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:84:00.0: PME# supported from D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.1: PCI bridge to [bus 84] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xac000000-0xadffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:85:00.0: [1022:145a] type 00 class 0x130000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:85:00.2: [1022:1456] type 00 class 0x108000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:85:00.2: reg 0x18: [mem 0xc0b00000-0xc0bfffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:85:00.2: reg 0x24: [mem 0xc0c00000-0xc0c01fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:07.1: PCI bridge to [bus 85] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.0: [1022:1455] type 00 class 0x130000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.1: [1022:1468] type 00 class 0x108000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.1: reg 0x18: [mem 0xc0900000-0xc09fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.1: reg 0x24: [mem 0xc0a00000-0xc0a01fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.2: [1022:7901] type 00 class 0x010601 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.2: reg 0x24: [mem 0xc0a02000-0xc0a02fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.2: PME# supported from D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:08.1: PCI bridge to [bus 86] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: on NUMA node 2 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC03] (domain 0000 [bus c0-ff]) Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:03: PCIe AER handled by firmware Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: platform does not support [SHPCHotplug] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:03: FADT indicates ASPM is unsupported, using BIOS configuration Jun 18 12:09:07 fir-md1-s1 kernel: acpi PNP0A08:03: host bridge window [mem 0x63dc0000000-0xffffffffffff window] ([0x80000000000-0xffffffffffff] ignored, not CPU addressable) Jun 18 12:09:07 fir-md1-s1 kernel: PCI host bridge to bus 0000:c0 Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [io 0xc000-0xffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [mem 0x90000000-0xaaffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [mem 0x63dc0000000-0x7ffffffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [bus c0-ff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:00.0: [1022:1450] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:00.2: [1022:1451] type 00 class 0x080600 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.1: [1022:1453] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:02.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:03.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:04.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:07.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:07.1: [1022:1454] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:07.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:08.0: [1022:1452] type 00 class 0x060000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:08.1: [1022:1454] type 01 class 0x060400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:08.1: PME# supported from D0 D3hot D3cold Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: [1000:005f] type 00 class 0x010400 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x10: [io 0xc000-0xc0ff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x14: [mem 0xa5500000-0xa550ffff 64bit] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x1c: [mem 0xa5400000-0xa54fffff 64bit] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: supports D1 D2 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.1: PCI bridge to [bus c1] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c2:00.0: [1022:145a] type 00 class 0x130000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c2:00.2: [1022:1456] type 00 class 0x108000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c2:00.2: reg 0x18: [mem 0xa5200000-0xa52fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c2:00.2: reg 0x24: [mem 0xa5300000-0xa5301fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:07.1: PCI bridge to [bus c2] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c3:00.0: [1022:1455] type 00 class 0x130000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c3:00.1: [1022:1468] type 00 class 0x108000 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c3:00.1: reg 0x18: [mem 0xa5000000-0xa50fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c3:00.1: reg 0x24: [mem 0xa5100000-0xa5101fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:08.1: PCI bridge to [bus c3] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c0: on NUMA node 3 Jun 18 12:09:07 fir-md1-s1 kernel: vgaarb: device added: PCI:0000:83:00.0,decodes=io+mem,owns=io+mem,locks=none Jun 18 12:09:07 fir-md1-s1 kernel: vgaarb: loaded Jun 18 12:09:07 fir-md1-s1 kernel: vgaarb: bridge control possible 0000:83:00.0 Jun 18 12:09:07 fir-md1-s1 kernel: SCSI subsystem initialized Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: bus type USB registered Jun 18 12:09:07 fir-md1-s1 kernel: usbcore: registered new interface driver usbfs Jun 18 12:09:07 fir-md1-s1 kernel: usbcore: registered new interface driver hub Jun 18 12:09:07 fir-md1-s1 kernel: usbcore: registered new device driver usb Jun 18 12:09:07 fir-md1-s1 kernel: EDAC MC: Ver: 3.0.0 Jun 18 12:09:07 fir-md1-s1 kernel: PCI: Using ACPI for IRQ routing Jun 18 12:09:07 fir-md1-s1 kernel: PCI: pci_cache_line_size set to 64 bytes Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x0008f000-0x0008ffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x4468f020-0x47ffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x446a8020-0x47ffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x446da020-0x47ffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x5b485020-0x5bffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x5c3e0000-0x5fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x6cacf000-0x6fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x107f380000-0x107fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x207ff80000-0x207fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x307ff80000-0x307fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x407ff80000-0x407fffffff] Jun 18 12:09:07 fir-md1-s1 kernel: NetLabel: Initializing Jun 18 12:09:07 fir-md1-s1 kernel: NetLabel: domain hash size = 128 Jun 18 12:09:07 fir-md1-s1 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Jun 18 12:09:07 fir-md1-s1 kernel: NetLabel: unlabeled traffic allowed by default Jun 18 12:09:07 fir-md1-s1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 Jun 18 12:09:07 fir-md1-s1 kernel: hpet0: 3 comparators, 32-bit 14.318180 MHz counter Jun 18 12:09:07 fir-md1-s1 kernel: Switched to clocksource hpet Jun 18 12:09:07 fir-md1-s1 kernel: pnp: PnP ACPI init Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: bus type PNP registered Jun 18 12:09:07 fir-md1-s1 kernel: system 00:00: [mem 0x80000000-0x8fffffff] has been reserved Jun 18 12:09:07 fir-md1-s1 kernel: system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) Jun 18 12:09:07 fir-md1-s1 kernel: pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active) Jun 18 12:09:07 fir-md1-s1 kernel: pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) Jun 18 12:09:07 fir-md1-s1 kernel: pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) Jun 18 12:09:07 fir-md1-s1 kernel: pnp: PnP ACPI: found 4 devices Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: bus type PNP unregistered Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:84:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: BAR 6: no space for [mem size 0x00040000 pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00040000 pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: PCI bridge to [bus 01] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [io 0x1000-0x1fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xf7500000-0xf75fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xe1000000-0xe11fffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:07.1: PCI bridge to [bus 02] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:08.1: PCI bridge to [bus 03] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 4 [io 0x0000-0x03af window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7 window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 6 [mem 0x000c0000-0x000c3fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 8 [mem 0x000c8000-0x000cbfff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 9 [mem 0x000cc000-0x000cffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 10 [mem 0x000d0000-0x000d3fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 11 [mem 0x000d4000-0x000d7fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 12 [mem 0x000d8000-0x000dbfff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 13 [mem 0x000dc000-0x000dffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 14 [mem 0x000e0000-0x000e3fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 15 [mem 0x000e4000-0x000e7fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 16 [mem 0x000e8000-0x000ebfff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 17 [mem 0x000ec000-0x000effff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 18 [mem 0x000f0000-0x000fffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 19 [io 0x0d00-0x3fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 20 [mem 0xe1000000-0xfebfffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:00: resource 21 [mem 0x10000000000-0x2bf3fffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:01: resource 0 [io 0x1000-0x1fff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:01: resource 1 [mem 0xf7500000-0xf75fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:01: resource 2 [mem 0xe1000000-0xe11fffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:02: resource 1 [mem 0xf7200000-0xf74fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:03: resource 1 [mem 0xf7000000-0xf71fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:07.1: PCI bridge to [bus 41] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:08.1: PCI bridge to [bus 42] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:40: resource 4 [io 0x4000-0x7fff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:40: resource 5 [mem 0xc6000000-0xe0ffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:40: resource 6 [mem 0x2bf40000000-0x47e7fffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:41: resource 1 [mem 0xdb200000-0xdb4fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:42: resource 1 [mem 0xdb000000-0xdb1fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.1: BAR 14: assigned [mem 0xab000000-0xab0fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.1: BAR 14: assigned [mem 0xab100000-0xab1fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: BAR 6: assigned [mem 0xab000000-0xab03ffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: BAR 6: assigned [mem 0xab040000-0xab07ffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.1: PCI bridge to [bus 81] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xab000000-0xab0fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xaf000000-0xaf0fffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:82:00.0: PCI bridge to [bus 83] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xae000000-0xaeffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.2: PCI bridge to [bus 82-83] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xae000000-0xaeffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:84:00.0: BAR 6: assigned [mem 0xab100000-0xab1fffff pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.1: PCI bridge to [bus 84] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xab100000-0xab1fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xac000000-0xadffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:07.1: PCI bridge to [bus 85] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:08.1: PCI bridge to [bus 86] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: resource 4 [io 0x03b0-0x03df window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: resource 5 [mem 0x000a0000-0x000bffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: resource 6 [io 0x8000-0xbfff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: resource 7 [mem 0xab000000-0xc5ffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:80: resource 8 [mem 0x47e80000000-0x63dbfffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:81: resource 1 [mem 0xab000000-0xab0fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:81: resource 2 [mem 0xaf000000-0xaf0fffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:82: resource 1 [mem 0xc0000000-0xc08fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:82: resource 2 [mem 0xae000000-0xaeffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:83: resource 1 [mem 0xc0000000-0xc08fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:83: resource 2 [mem 0xae000000-0xaeffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:84: resource 1 [mem 0xab100000-0xab1fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:84: resource 2 [mem 0xac000000-0xadffffff 64bit pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:85: resource 1 [mem 0xc0b00000-0xc0cfffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:86: resource 1 [mem 0xc0900000-0xc0afffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: BAR 6: no space for [mem size 0x00100000 pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.1: PCI bridge to [bus c1] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:07.1: PCI bridge to [bus c2] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:08.1: PCI bridge to [bus c3] Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c0: resource 4 [io 0xc000-0xffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c0: resource 5 [mem 0x90000000-0xaaffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c0: resource 6 [mem 0x63dc0000000-0x7ffffffffff window] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c1: resource 0 [io 0xc000-0xcfff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c1: resource 1 [mem 0xa5400000-0xa55fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c2: resource 1 [mem 0xa5200000-0xa53fffff] Jun 18 12:09:07 fir-md1-s1 kernel: pci_bus 0000:c3: resource 1 [mem 0xa5000000-0xa51fffff] Jun 18 12:09:07 fir-md1-s1 kernel: NET: Registered protocol family 2 Jun 18 12:09:07 fir-md1-s1 kernel: TCP established hash table entries: 524288 (order: 10, 4194304 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: TCP: Hash tables configured (established 524288 bind 65536) Jun 18 12:09:07 fir-md1-s1 kernel: TCP: reno registered Jun 18 12:09:07 fir-md1-s1 kernel: UDP hash table entries: 65536 (order: 9, 2097152 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: NET: Registered protocol family 1 Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:83:00.0: Boot video device Jun 18 12:09:07 fir-md1-s1 kernel: PCI: CLS 32 bytes, default 64 Jun 18 12:09:07 fir-md1-s1 kernel: Unpacking initramfs... Jun 18 12:09:07 fir-md1-s1 kernel: Freeing initrd memory: 19788k freed Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:01.0 to group 0 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:02.0 to group 1 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:03.0 to group 2 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:03.1 to group 2 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:04.0 to group 3 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:07.0 to group 4 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:07.1 to group 4 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:08.0 to group 5 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:08.1 to group 5 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:14.0 to group 6 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:14.3 to group 6 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.0 to group 7 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.1 to group 7 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.2 to group 7 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.3 to group 7 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.4 to group 7 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.5 to group 7 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.6 to group 7 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.7 to group 7 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.0 to group 8 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.1 to group 8 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.2 to group 8 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.3 to group 8 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.4 to group 8 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.5 to group 8 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.6 to group 8 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.7 to group 8 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.0 to group 9 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.1 to group 9 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.2 to group 9 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.3 to group 9 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.4 to group 9 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.5 to group 9 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.6 to group 9 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.7 to group 9 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.0 to group 10 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.1 to group 10 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.2 to group 10 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.3 to group 10 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.4 to group 10 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.5 to group 10 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.6 to group 10 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.7 to group 10 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:01:00.0 to group 2 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.0 to group 4 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.2 to group 4 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.3 to group 4 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:03:00.0 to group 5 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:03:00.1 to group 5 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:40:01.0 to group 11 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:40:02.0 to group 12 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:40:03.0 to group 13 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:40:04.0 to group 14 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:40:07.0 to group 15 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:40:07.1 to group 15 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:40:08.0 to group 16 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:40:08.1 to group 16 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.0 to group 15 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.2 to group 15 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.3 to group 15 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:42:00.0 to group 16 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:42:00.1 to group 16 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.0 to group 17 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.1 to group 17 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.2 to group 17 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:02.0 to group 18 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:03.0 to group 19 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:03.1 to group 19 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:04.0 to group 20 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:07.0 to group 21 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:07.1 to group 21 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:08.0 to group 22 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:80:08.1 to group 22 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:81:00.0 to group 17 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:81:00.1 to group 17 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:82:00.0 to group 17 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:83:00.0 to group 17 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:84:00.0 to group 19 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:85:00.0 to group 21 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:85:00.2 to group 21 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.0 to group 22 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.1 to group 22 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.2 to group 22 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:01.0 to group 23 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:01.1 to group 23 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:02.0 to group 24 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:03.0 to group 25 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:04.0 to group 26 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:07.0 to group 27 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:07.1 to group 27 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:08.0 to group 28 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c0:08.1 to group 28 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c1:00.0 to group 23 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c2:00.0 to group 27 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c2:00.2 to group 27 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c3:00.0 to group 28 Jun 18 12:09:07 fir-md1-s1 kernel: iommu: Adding device 0000:c3:00.1 to group 28 Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Jun 18 12:09:07 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Jun 18 12:09:07 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:80:00.2 cap 0x40 Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Jun 18 12:09:07 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:c0:00.2 cap 0x40 Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Jun 18 12:09:07 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Interrupt remapping enabled Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: virtual APIC enabled Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:00:00.2: irq 26 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:40:00.2: irq 27 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:80:00.2: irq 28 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c0:00.2: irq 29 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: AMD-Vi: Lazy IO/TLB flushing enabled Jun 18 12:09:07 fir-md1-s1 kernel: perf: AMD NB counters detected Jun 18 12:09:07 fir-md1-s1 kernel: perf: AMD LLC counters detected Jun 18 12:09:07 fir-md1-s1 kernel: sha1_ssse3: Using SHA-NI optimized SHA-1 implementation Jun 18 12:09:07 fir-md1-s1 kernel: sha256_ssse3: Using SHA-256-NI optimized SHA-256 implementation Jun 18 12:09:07 fir-md1-s1 kernel: futex hash table entries: 32768 (order: 9, 2097152 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: Initialise system trusted keyring Jun 18 12:09:07 fir-md1-s1 kernel: audit: initializing netlink socket (disabled) Jun 18 12:09:07 fir-md1-s1 kernel: type=2000 audit(1560884939.538:1): initialized Jun 18 12:09:07 fir-md1-s1 kernel: HugeTLB registered 1 GB page size, pre-allocated 0 pages Jun 18 12:09:07 fir-md1-s1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages Jun 18 12:09:07 fir-md1-s1 kernel: zpool: loaded Jun 18 12:09:07 fir-md1-s1 kernel: zbud: loaded Jun 18 12:09:07 fir-md1-s1 kernel: VFS: Disk quotas dquot_6.5.2 Jun 18 12:09:07 fir-md1-s1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Jun 18 12:09:07 fir-md1-s1 kernel: msgmni has been set to 32768 Jun 18 12:09:07 fir-md1-s1 kernel: Key type big_key registered Jun 18 12:09:07 fir-md1-s1 kernel: SELinux: Registering netfilter hooks Jun 18 12:09:07 fir-md1-s1 kernel: NET: Registered protocol family 38 Jun 18 12:09:07 fir-md1-s1 kernel: Key type asymmetric registered Jun 18 12:09:07 fir-md1-s1 kernel: Asymmetric key parser 'x509' registered Jun 18 12:09:07 fir-md1-s1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) Jun 18 12:09:07 fir-md1-s1 kernel: io scheduler noop registered Jun 18 12:09:07 fir-md1-s1 kernel: io scheduler deadline registered (default) Jun 18 12:09:07 fir-md1-s1 kernel: io scheduler cfq registered Jun 18 12:09:07 fir-md1-s1 kernel: io scheduler mq-deadline registered Jun 18 12:09:07 fir-md1-s1 kernel: io scheduler kyber registered Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:00:03.1: irq 30 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:00:07.1: irq 31 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:00:08.1: irq 33 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:40:07.1: irq 34 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:40:08.1: irq 36 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:01.1: irq 37 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:01.2: irq 38 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:03.1: irq 39 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:07.1: irq 41 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:08.1: irq 43 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:c0:01.1: irq 44 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:c0:07.1: irq 46 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:c0:08.1: irq 48 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:01:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:00:03.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:00:07.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.2: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:02:00.3: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:00:07.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:00:08.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:03:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:03:00.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:00:08.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:40:07.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.2: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:41:00.3: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:40:07.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:40:08.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:42:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:42:00.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:40:08.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:01.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:81:00.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:80:01.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:01.2: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:82:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:83:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:80:01.2:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:03.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:84:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:80:03.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:07.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:85:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:85:00.2: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:80:07.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:80:08.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:86:00.2: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:80:08.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:c0:01.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c1:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:c0:01.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:c0:07.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c2:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c2:00.2: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:c0:07.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pcieport 0000:c0:08.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c3:00.0: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pci 0000:c3:00.1: Signaling PME through PCIe PME interrupt Jun 18 12:09:07 fir-md1-s1 kernel: pcie_pme 0000:c0:08.1:pcie001: service driver pcie_pme loaded Jun 18 12:09:07 fir-md1-s1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Jun 18 12:09:07 fir-md1-s1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Jun 18 12:09:07 fir-md1-s1 kernel: shpchp 0000:82:00.0: Cannot get control of SHPC hotplug Jun 18 12:09:07 fir-md1-s1 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Jun 18 12:09:07 fir-md1-s1 kernel: efifb: probing for efifb Jun 18 12:09:07 fir-md1-s1 kernel: efifb: framebuffer at 0xae000000, mapped to 0xffffb01219800000, using 3072k, total 3072k Jun 18 12:09:07 fir-md1-s1 kernel: efifb: mode is 1024x768x32, linelength=4096, pages=1 Jun 18 12:09:07 fir-md1-s1 kernel: efifb: scrolling: redraw Jun 18 12:09:07 fir-md1-s1 kernel: efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0 Jun 18 12:09:07 fir-md1-s1 kernel: Console: switching to colour frame buffer device 128x48 Jun 18 12:09:07 fir-md1-s1 kernel: fb0: EFI VGA frame buffer device Jun 18 12:09:07 fir-md1-s1 kernel: input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Power Button [PWRB] Jun 18 12:09:07 fir-md1-s1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 Jun 18 12:09:07 fir-md1-s1 kernel: ACPI: Power Button [PWRF] Jun 18 12:09:07 fir-md1-s1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. Jun 18 12:09:07 fir-md1-s1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Jun 18 12:09:07 fir-md1-s1 kernel: 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Jun 18 12:09:07 fir-md1-s1 kernel: 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jun 18 12:09:07 fir-md1-s1 kernel: Non-volatile memory driver v1.3 Jun 18 12:09:07 fir-md1-s1 kernel: Linux agpgart interface v0.103 Jun 18 12:09:07 fir-md1-s1 kernel: crash memory driver: version 1.1 Jun 18 12:09:07 fir-md1-s1 kernel: rdac: device handler registered Jun 18 12:09:07 fir-md1-s1 kernel: hp_sw: device handler registered Jun 18 12:09:07 fir-md1-s1 kernel: emc: device handler registered Jun 18 12:09:07 fir-md1-s1 kernel: alua: device handler registered Jun 18 12:09:07 fir-md1-s1 kernel: libphy: Fixed MDIO Bus: probed Jun 18 12:09:07 fir-md1-s1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Jun 18 12:09:07 fir-md1-s1 kernel: ehci-pci: EHCI PCI platform driver Jun 18 12:09:07 fir-md1-s1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Jun 18 12:09:07 fir-md1-s1 kernel: ohci-pci: OHCI PCI platform driver Jun 18 12:09:07 fir-md1-s1 kernel: uhci_hcd: USB Universal Host Controller Interface driver Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: xHCI Host Controller Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 1 Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 50 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 51 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 52 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 53 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 54 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 55 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 56 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 57 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb1: Product: xHCI Host Controller Jun 18 12:09:07 fir-md1-s1 kernel: usb usb1: Manufacturer: Linux 3.10.0-957.1.3.el7_lustre.x86_64 xhci-hcd Jun 18 12:09:07 fir-md1-s1 kernel: usb usb1: SerialNumber: 0000:02:00.3 Jun 18 12:09:07 fir-md1-s1 kernel: hub 1-0:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 1-0:1.0: 2 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: xHCI Host Controller Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 2 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. Jun 18 12:09:07 fir-md1-s1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb2: Product: xHCI Host Controller Jun 18 12:09:07 fir-md1-s1 kernel: usb usb2: Manufacturer: Linux 3.10.0-957.1.3.el7_lustre.x86_64 xhci-hcd Jun 18 12:09:07 fir-md1-s1 kernel: usb usb2: SerialNumber: 0000:02:00.3 Jun 18 12:09:07 fir-md1-s1 kernel: hub 2-0:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 2-0:1.0: 2 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: xHCI Host Controller Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 3 Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 59 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 60 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 61 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 62 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 63 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 64 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 65 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 66 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb3: Product: xHCI Host Controller Jun 18 12:09:07 fir-md1-s1 kernel: usb usb3: Manufacturer: Linux 3.10.0-957.1.3.el7_lustre.x86_64 xhci-hcd Jun 18 12:09:07 fir-md1-s1 kernel: usb usb3: SerialNumber: 0000:41:00.3 Jun 18 12:09:07 fir-md1-s1 kernel: hub 3-0:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 3-0:1.0: 2 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: xHCI Host Controller Jun 18 12:09:07 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 4 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb4: We don't know the algorithms for LPM for this host, disabling LPM. Jun 18 12:09:07 fir-md1-s1 kernel: usb usb4: New USB device found, idVendor=1d6b, idProduct=0003 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jun 18 12:09:07 fir-md1-s1 kernel: usb usb4: Product: xHCI Host Controller Jun 18 12:09:07 fir-md1-s1 kernel: usb usb4: Manufacturer: Linux 3.10.0-957.1.3.el7_lustre.x86_64 xhci-hcd Jun 18 12:09:07 fir-md1-s1 kernel: usb usb4: SerialNumber: 0000:41:00.3 Jun 18 12:09:07 fir-md1-s1 kernel: hub 4-0:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 4-0:1.0: 2 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: usbcore: registered new interface driver usbserial_generic Jun 18 12:09:07 fir-md1-s1 kernel: usbserial: USB Serial support registered for generic Jun 18 12:09:07 fir-md1-s1 kernel: i8042: PNP: No PS/2 controller found. Probing ports directly. Jun 18 12:09:07 fir-md1-s1 kernel: usb 1-1: new high-speed USB device number 2 using xhci_hcd Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1: new high-speed USB device number 2 using xhci_hcd Jun 18 12:09:07 fir-md1-s1 kernel: usb 1-1: New USB device found, idVendor=0424, idProduct=2744 Jun 18 12:09:07 fir-md1-s1 kernel: usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Jun 18 12:09:07 fir-md1-s1 kernel: usb 1-1: Product: USB2734 Jun 18 12:09:07 fir-md1-s1 kernel: usb 1-1: Manufacturer: Microchip Tech Jun 18 12:09:07 fir-md1-s1 kernel: hub 1-1:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 1-1:1.0: 4 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1: New USB device found, idVendor=1604, idProduct=10c0 Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Jun 18 12:09:07 fir-md1-s1 kernel: hub 3-1:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 3-1:1.0: 4 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: usb 2-1: New USB device found, idVendor=0424, idProduct=5744 Jun 18 12:09:07 fir-md1-s1 kernel: usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=0 Jun 18 12:09:07 fir-md1-s1 kernel: usb 2-1: Product: USB5734 Jun 18 12:09:07 fir-md1-s1 kernel: usb 2-1: Manufacturer: Microchip Tech Jun 18 12:09:07 fir-md1-s1 kernel: hub 2-1:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 2-1:1.0: 4 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: usb: port power management may be unreliable Jun 18 12:09:07 fir-md1-s1 kernel: i8042: No controller found Jun 18 12:09:07 fir-md1-s1 kernel: tsc: Refined TSC clocksource calibration: 1996.249 MHz Jun 18 12:09:07 fir-md1-s1 kernel: mousedev: PS/2 mouse device common for all mice Jun 18 12:09:07 fir-md1-s1 kernel: rtc_cmos 00:01: RTC can wake from S4 Jun 18 12:09:07 fir-md1-s1 kernel: rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 Jun 18 12:09:07 fir-md1-s1 kernel: rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs Jun 18 12:09:07 fir-md1-s1 kernel: cpuidle: using governor menu Jun 18 12:09:07 fir-md1-s1 kernel: EFI Variables Facility v0.08 2004-May-17 Jun 18 12:09:07 fir-md1-s1 kernel: hidraw: raw HID events driver (C) Jiri Kosina Jun 18 12:09:07 fir-md1-s1 kernel: usbcore: registered new interface driver usbhid Jun 18 12:09:07 fir-md1-s1 kernel: usbhid: USB HID core driver Jun 18 12:09:07 fir-md1-s1 kernel: drop_monitor: Initializing network drop monitor service Jun 18 12:09:07 fir-md1-s1 kernel: TCP: cubic registered Jun 18 12:09:07 fir-md1-s1 kernel: Initializing XFRM netlink socket Jun 18 12:09:07 fir-md1-s1 kernel: NET: Registered protocol family 10 Jun 18 12:09:07 fir-md1-s1 kernel: NET: Registered protocol family 17 Jun 18 12:09:07 fir-md1-s1 kernel: mpls_gso: MPLS GSO support Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU0: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU1: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU2: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU3: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU4: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU5: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1.1: new high-speed USB device number 3 using xhci_hcd Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU6: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU7: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: Switched to clocksource tsc Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU8: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU9: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU10: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU11: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU12: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU13: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU14: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU15: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU16: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU17: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU18: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU19: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU20: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU21: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU22: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU23: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU24: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU25: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU26: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU27: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU28: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU29: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU30: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU31: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU32: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU33: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU34: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU35: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU36: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU37: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU38: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU39: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU40: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU41: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU42: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU43: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU44: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU45: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU46: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: CPU47: patch_level=0x08001227 Jun 18 12:09:07 fir-md1-s1 kernel: microcode: Microcode Update Driver: v2.01 , Peter Oruba Jun 18 12:09:07 fir-md1-s1 kernel: PM: Hibernation image not present or could not be loaded. Jun 18 12:09:07 fir-md1-s1 kernel: Loading compiled-in X.509 certificates Jun 18 12:09:07 fir-md1-s1 kernel: Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80' Jun 18 12:09:07 fir-md1-s1 kernel: Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8' Jun 18 12:09:07 fir-md1-s1 kernel: Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: 26463bf7b35aa6e910b2216d61318fa5ff5b7954' Jun 18 12:09:07 fir-md1-s1 kernel: registered taskstats version 1 Jun 18 12:09:07 fir-md1-s1 kernel: Key type trusted registered Jun 18 12:09:07 fir-md1-s1 kernel: Key type encrypted registered Jun 18 12:09:07 fir-md1-s1 kernel: IMA: No TPM chip found, activating TPM-bypass! (rc=-19) Jun 18 12:09:07 fir-md1-s1 kernel: Magic number: 7:983:192 Jun 18 12:09:07 fir-md1-s1 kernel: acpi device:1e: hash matches Jun 18 12:09:07 fir-md1-s1 kernel: memory memory1550: hash matches Jun 18 12:09:07 fir-md1-s1 kernel: memory memory763: hash matches Jun 18 12:09:07 fir-md1-s1 kernel: rtc_cmos 00:01: setting system clock to 2019-06-18 19:09:06 UTC (1560884946) Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1.1: New USB device found, idVendor=1604, idProduct=10c0 Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Jun 18 12:09:07 fir-md1-s1 kernel: hub 3-1.1:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 3-1.1:1.0: 4 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1.4: new high-speed USB device number 4 using xhci_hcd Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1.4: New USB device found, idVendor=1604, idProduct=10c0 Jun 18 12:09:07 fir-md1-s1 kernel: usb 3-1.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Jun 18 12:09:07 fir-md1-s1 kernel: hub 3-1.4:1.0: USB hub found Jun 18 12:09:07 fir-md1-s1 kernel: hub 3-1.4:1.0: 4 ports detected Jun 18 12:09:07 fir-md1-s1 kernel: Freeing unused kernel memory: 1876k freed Jun 18 12:09:07 fir-md1-s1 kernel: Write protecting the kernel read-only data: 12288k Jun 18 12:09:07 fir-md1-s1 kernel: Freeing unused kernel memory: 516k freed Jun 18 12:09:07 fir-md1-s1 kernel: Freeing unused kernel memory: 600k freed Jun 18 12:09:07 fir-md1-s1 systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Jun 18 12:09:07 fir-md1-s1 systemd[1]: Detected architecture x86-64. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Running in initial RAM disk. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Set hostname to . Jun 18 12:09:07 fir-md1-s1 systemd[1]: Reached target Swap. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Reached target Timers. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Created slice Root Slice. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Listening on udev Control Socket. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Created slice System Slice. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Reached target Slices. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Listening on Journal Socket. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Starting Load Kernel Modules... Jun 18 12:09:07 fir-md1-s1 systemd[1]: Starting Create list of required static device nodes for the current kernel... Jun 18 12:09:07 fir-md1-s1 systemd[1]: Starting Journal Service... Jun 18 12:09:07 fir-md1-s1 systemd[1]: Starting dracut cmdline hook... Jun 18 12:09:07 fir-md1-s1 systemd[1]: Starting Setup Virtual Console... Jun 18 12:09:07 fir-md1-s1 systemd[1]: Listening on udev Kernel Socket. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Reached target Sockets. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Reached target Local File Systems. Jun 18 12:09:07 fir-md1-s1 systemd[1]: Started Journal Service. Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas: loading out-of-tree module taints kernel. Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas: module verification failed: signature and/or required key missing - tainting kernel Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas version 27.00.00.00 loaded Jun 18 12:09:07 fir-md1-s1 kernel: pps_core: LinuxPPS API ver. 1 registered Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (263565264 kB) Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: IOC Number : 0 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 68 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 69 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 70 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 71 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 72 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 73 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 74 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 75 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 76 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 77 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 78 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 79 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 80 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 81 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 82 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 83 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 84 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 85 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 86 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 87 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 88 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 89 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 90 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 91 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 92 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 93 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 94 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 95 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 96 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 97 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 98 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 99 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 100 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 101 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 102 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 103 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 104 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 105 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 106 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 107 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 108 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 109 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 110 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 111 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 112 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 113 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 114 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas 0000:01:00.0: irq 115 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 68 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 69 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 70 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 71 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 72 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 73 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 74 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 75 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 76 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 77 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 78 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 79 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 80 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 81 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 82 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 83 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 84 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 85 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 86 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 87 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 88 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 89 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 90 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 91 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 92 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 93 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 94 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 95 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 96 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 97 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 98 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 99 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 100 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 101 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 102 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 103 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 104 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 105 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 106 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 107 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 108 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 109 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 110 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 111 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 112 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 113 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 114 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 115 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: iomem(0x00000000e1000000), mapped(0xffffb0121a000000), size(1048576) Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: ioport(0x0000000000001000), size(256) Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: IOC Number : 0 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k Jun 18 12:09:07 fir-md1-s1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti Jun 18 12:09:07 fir-md1-s1 kernel: megasas: 07.705.02.00-rh1 Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: FW now in Ready state Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: 64 bit DMA mask and 32 bit consistent mask Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 117 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 118 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 119 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 120 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 121 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 122 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 123 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 124 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 125 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 126 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 127 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 128 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 129 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 130 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 131 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 132 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 133 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 134 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 135 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 136 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 137 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 138 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 139 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 140 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 141 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 142 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 143 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 144 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 145 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 146 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 147 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 148 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 149 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 150 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 151 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 152 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 153 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 154 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 155 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 156 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 157 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 158 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 159 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 160 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 161 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 162 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 163 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 164 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware supports msix : (96) Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: current msix/online cpus : (48/48) Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: RDPQ mode : (disabled) Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 237 Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Configured max firmware commands: 927 Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: FW supports sync cache : No Jun 18 12:09:07 fir-md1-s1 kernel: PTP clock support registered Jun 18 12:09:07 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: Allocated physical memory: size(38831 kB) Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: Current Controller Queue Depth(7564), Max Controller Queue Depth(7680) Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: Scatter Gather Elements per IO(128) Jun 18 12:09:07 fir-md1-s1 kernel: libata version 3.00 loaded. Jun 18 12:09:07 fir-md1-s1 kernel: Compat-mlnx-ofed backport release: b4fdfac Jun 18 12:09:07 fir-md1-s1 kernel: Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git b4fdfac Jun 18 12:09:07 fir-md1-s1 kernel: compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: FW Package Version(08.00.00.00) Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: SAS3616: FWVersion(08.00.00.00), ChipRevision(0x02), BiosVersion(00.00.00.00) Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: Protocol=(Initiator,Target,NVMe), Capabilities=(TLR,EEDP,Diag Trace Buffer,Task Set Full,NCQ) Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: : host protection capabilities enabled DIF1 DIF2 DIF3 Jun 18 12:09:07 fir-md1-s1 kernel: scsi host0: Fusion MPT SAS Host Jun 18 12:09:07 fir-md1-s1 kernel: mpt3sas_cm0: sending port enable !! Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Init cmd return status SUCCESS for SCSI host 1 Jun 18 12:09:07 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware type : Legacy(64 VD) firmware Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: controller type : iMR(0MB) Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Online Controller Reset(OCR) : Enabled Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Secure JBOD support : No Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: NVMe passthru support : No Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: INIT adapter done Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Jbod map is not supported megasas_setup_jbod_map 5146 Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: pci id : (0x1000)/(0x005f)/(0x1028)/(0x1f4b) Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: unevenspan support : yes Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware crash dump : no Jun 18 12:09:07 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: jbod sync map : no Jun 18 12:09:07 fir-md1-s1 kernel: scsi host1: Avago SAS based MegaRAID driver Jun 18 12:09:07 fir-md1-s1 kernel: scsi 1:2:0:0: Direct-Access DELL PERC H330 Mini 4.29 PQ: 0 ANSI: 5 Jun 18 12:09:07 fir-md1-s1 kernel: tg3.c:v3.137 (May 11, 2014) Jun 18 12:09:07 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: version 3.0 Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 167 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 168 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 169 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 170 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 171 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 172 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 173 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 174 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 175 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 176 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 177 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 178 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 179 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 180 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 181 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 182 for MSI/MSI-X Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode Jun 18 12:09:07 fir-md1-s1 kernel: ahci 0000:86:00.2: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part Jun 18 12:09:07 fir-md1-s1 kernel: scsi host2: ahci Jun 18 12:09:07 fir-md1-s1 kernel: ata1: SATA max UDMA/133 abar m4096@0xc0a02000 port 0xc0a02100 irq 167 Jun 18 12:09:07 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:34:4a:7d Jun 18 12:09:07 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Jun 18 12:09:07 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Jun 18 12:09:07 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] Jun 18 12:09:08 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: firmware version: 12.24.1000 Jun 18 12:09:08 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: 126.016 Gb/s available PCIe bandwidth (8 GT/s x16 link) Jun 18 12:09:08 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:34:4a:7e Jun 18 12:09:08 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Jun 18 12:09:08 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Jun 18 12:09:08 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] Jun 18 12:09:08 fir-md1-s1 kernel: ata1: SATA link down (SStatus 0 SControl 300) Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 185 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 186 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 187 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 188 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 189 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 190 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 191 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 192 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 193 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 194 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 195 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 196 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 197 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 198 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 199 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 200 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 201 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 202 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 203 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 204 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 205 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 206 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 207 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 208 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 209 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 210 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 211 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 212 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 213 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 214 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 215 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 216 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 217 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 218 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 219 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 220 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 221 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 222 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 223 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 224 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 225 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 226 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 227 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 228 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 229 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 230 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 231 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 232 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 233 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 234 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 235 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: irq 236 for MSI/MSI-X Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: Port module event: module 0, Cable plugged Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: hba_port entry: ffff8f3536ddec00, port: 255 is added to hba_port list Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00db90c00), phys(17) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0011), sas_address(0x510600b00db90c00), phy(16) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0011), retries(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0011), lun(0) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:0:0: Enclosure LSI virtualSES 02 PQ: 0 ANSI: 6 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:0:0: set ignore_delay_remove for handle(0x0011) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:0:0: SES: handle(0x0011), sas_addr(0x510600b00db90c00), phy(16), device_name(0x510600b00db90c00) Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: FW Tracer Owner Jun 18 12:09:10 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:0:0: enclosure logical id(0x500605b00db90c00), slot(16) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:0:0: enclosure level(0x0000), connector name( C3 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:0:0: serial_number(500605B00DB90C00) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:0:0: qdepth(1), tagged(0), simple(0), ordered(0), scsi_level(7), cmd_que(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: log_info(0x31200206): originator(PL), code(0x20), sub_code(0x0206) Jun 18 12:09:10 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:09:10 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_ib: Mellanox Connect-IB Infiniband driver v4.5-1.0.1 Jun 18 12:09:10 fir-md1-s1 kernel: mlx5_ib: Mellanox Connect-IB Infiniband driver v4.5-1.0.1 Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0012), sas_address(0x500a0984db2fa920), phy(8) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0012), retries(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0012), retries(1) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0012), lun(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0012), sas_address(0x500a0984db2fa920), phy(8) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0012), retries(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0012), lun(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0012), sas_address(0x500a0984db2fa920), phy(8) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0012), retries(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0012), lun(0) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:0: SSP: handle(0x0012), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:0: enclosure logical id(0x500605b00db90c00), slot(5) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:0: enclosure level(0x0000), connector name( C1 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:0: serial_number(021815000354 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:1: SSP: handle(0x0012), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Jun 18 12:09:10 fir-md1-s1 kernel: random: crng init done Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:1: enclosure logical id(0x500605b00db90c00), slot(5) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:1: enclosure level(0x0000), connector name( C1 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:1: serial_number(021815000354 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:1: Mode parameters changed Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:2: SSP: handle(0x0012), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:2: enclosure logical id(0x500605b00db90c00), slot(5) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:2: enclosure level(0x0000), connector name( C1 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:2: serial_number(021815000354 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:2: Mode parameters changed Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:31: SSP: handle(0x0012), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:31: enclosure logical id(0x500605b00db90c00), slot(5) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:31: enclosure level(0x0000), connector name( C1 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:31: serial_number(021815000354 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:1:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0013), sas_address(0x500a0984dfa1fa20), phy(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0013), retries(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0013), retries(1) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0013), lun(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0013), sas_address(0x500a0984dfa1fa20), phy(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0013), retries(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0013), lun(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0013), sas_address(0x500a0984dfa1fa20), phy(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0013), retries(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0013), lun(0) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:0: SSP: handle(0x0013), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:0: enclosure logical id(0x500605b00db90c00), slot(13) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:0: enclosure level(0x0000), connector name( C3 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:0: serial_number(021825001369 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:1: SSP: handle(0x0013), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:1: enclosure logical id(0x500605b00db90c00), slot(13) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:1: enclosure level(0x0000), connector name( C3 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:1: serial_number(021825001369 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:1: Mode parameters changed Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:31: SSP: handle(0x0013), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:31: enclosure logical id(0x500605b00db90c00), slot(13) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:31: enclosure level(0x0000), connector name( C3 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:31: serial_number(021825001369 ) Jun 18 12:09:10 fir-md1-s1 kernel: scsi 0:0:2:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0014), sas_address(0x500a0984da0f9b14), phy(12) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0014), retries(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0014), retries(1) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0014), lun(0) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0014), sas_address(0x500a0984da0f9b14), phy(12) Jun 18 12:09:10 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0014), retries(0) Jun 18 12:09:11 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0014), lun(0) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:0: SSP: handle(0x0014), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:0: enclosure logical id(0x500605b00db90c00), slot(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:0: enclosure level(0x0000), connector name( C0 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:0: serial_number(021812047179 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:1: SSP: handle(0x0014), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:1: enclosure logical id(0x500605b00db90c00), slot(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:1: enclosure level(0x0000), connector name( C0 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:1: serial_number(021812047179 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:2: SSP: handle(0x0014), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:2: enclosure logical id(0x500605b00db90c00), slot(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:2: enclosure level(0x0000), connector name( C0 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:2: serial_number(021812047179 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:31: SSP: handle(0x0014), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:31: enclosure logical id(0x500605b00db90c00), slot(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:31: enclosure level(0x0000), connector name( C0 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:31: serial_number(021812047179 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:3:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:11 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0015), sas_address(0x500a0984dfa20c14), phy(4) Jun 18 12:09:11 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0015), retries(0) Jun 18 12:09:11 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0015), retries(1) Jun 18 12:09:11 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0015), lun(0) Jun 18 12:09:11 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0015), sas_address(0x500a0984dfa20c14), phy(4) Jun 18 12:09:11 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0015), retries(0) Jun 18 12:09:11 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0015), lun(0) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:0: SSP: handle(0x0015), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:0: enclosure logical id(0x500605b00db90c00), slot(9) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:0: enclosure level(0x0000), connector name( C2 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:0: serial_number(021825001558 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:1: SSP: handle(0x0015), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:1: enclosure logical id(0x500605b00db90c00), slot(9) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:1: enclosure level(0x0000), connector name( C2 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:1: serial_number(021825001558 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:31: SSP: handle(0x0015), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:31: enclosure logical id(0x500605b00db90c00), slot(9) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:31: enclosure level(0x0000), connector name( C2 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:31: serial_number(021825001558 ) Jun 18 12:09:11 fir-md1-s1 kernel: scsi 0:0:4:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Jun 18 12:09:16 fir-md1-s1 kernel: mpt3sas_cm0: port enable: SUCCESS Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:1:0: rdac: LUN 0 (IOSHIP) (owned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:0: [sda] 926167040 512-byte logical blocks: (474 GB/441 GiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:0: [sda] 4096-byte physical blocks Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:1:1: rdac: LUN 1 (IOSHIP) (unowned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:0: [sda] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:1: [sdb] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:0: [sda] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:1:2: rdac: LUN 2 (IOSHIP) (owned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:2: [sdc] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:2:0: rdac: LUN 0 (IOSHIP) (owned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:2: [sdc] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:2: [sdc] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:0: [sdd] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:2: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:2:1: rdac: LUN 1 (IOSHIP) (unowned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:0: [sdd] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:0: [sdd] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:1: [sde] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:3:0: rdac: LUN 0 (IOSHIP) (unowned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:0: [sdf] 926167040 512-byte logical blocks: (474 GB/441 GiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:0: [sdf] 4096-byte physical blocks Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:1: [sde] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:1: [sde] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:1: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:3:1: rdac: LUN 1 (IOSHIP) (owned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:0: [sdf] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:0: [sdf] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:1: [sdg] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:0: [sda] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:3:2: rdac: LUN 2 (IOSHIP) (unowned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:1: [sdg] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:1: [sdg] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:2: [sdc] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:2: [sdh] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:1: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:4:0: rdac: LUN 0 (IOSHIP) (unowned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:2: [sdh] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:2: [sdh] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:0: [sdi] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:2: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:0: [sdd] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: scsi 0:0:4:1: rdac: LUN 1 (IOSHIP) (owned) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:0: [sdi] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:0: [sdi] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:1: [sdj] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: sd 1:2:0:0: [sdk] 233308160 512-byte logical blocks: (119 GB/111 GiB) Jun 18 12:09:16 fir-md1-s1 kernel: sd 1:2:0:0: [sdk] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 1:2:0:0: [sdk] Mode Sense: 1f 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 1:2:0:0: [sdk] Write cache: disabled, read cache: disabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:1: [sdj] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:1: [sdj] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:1: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:1: [sdg] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:0: [sdf] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sdk: sdk1 sdk2 sdk3 Jun 18 12:09:16 fir-md1-s1 kernel: sd 1:2:0:0: [sdk] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:2:1: [sde] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:3:2: [sdh] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:1: [sdj] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:4:0: [sdi] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:1: [sdb] Write Protect is off Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:1: [sdb] Mode Sense: 83 00 10 08 Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA Jun 18 12:09:16 fir-md1-s1 kernel: sd 0:0:1:1: [sdb] Attached SCSI disk Jun 18 12:09:16 fir-md1-s1 kernel: EXT4-fs (sdk2): mounted filesystem with ordered data mode. Opts: (null) Jun 18 12:09:17 fir-md1-s1 systemd-journald[357]: Received SIGTERM from PID 1 (systemd). Jun 18 12:09:17 fir-md1-s1 kernel: SELinux: Disabled at runtime. Jun 18 12:09:17 fir-md1-s1 kernel: SELinux: Unregistering netfilter hooks Jun 18 12:09:17 fir-md1-s1 kernel: type=1404 audit(1560884956.963:2): selinux=0 auid=4294967295 ses=4294967295 Jun 18 12:09:17 fir-md1-s1 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team Jun 18 12:09:17 fir-md1-s1 systemd[1]: Inserted module 'ip_tables' Jun 18 12:09:17 fir-md1-s1 kernel: EXT4-fs (sdk2): re-mounted. Opts: (null) Jun 18 12:09:17 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:09:17 fir-md1-s1 kernel: knem 1.1.3.90mlnx1: initialized Jun 18 12:09:17 fir-md1-s1 kernel: ACPI Error: No handler for Region [SYSI] (ffff8f35e9e82b40) [IPMI] (20130517/evregion-162) Jun 18 12:09:17 fir-md1-s1 kernel: ACPI Error: Jun 18 12:09:17 fir-md1-s1 kernel: Region IPMI (ID=7) has no handler Jun 18 12:09:17 fir-md1-s1 kernel: (20130517/exfldio-305) Jun 18 12:09:17 fir-md1-s1 kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff8f15e9e7b5a0), AE_NOT_EXIST (20130517/psparse-536) Jun 18 12:09:17 fir-md1-s1 kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff8f15e9e7b500), AE_NOT_EXIST (20130517/psparse-536) Jun 18 12:09:17 fir-md1-s1 kernel: ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753) Jun 18 12:09:17 fir-md1-s1 kernel: ipmi message handler version 39.2 Jun 18 12:09:17 fir-md1-s1 kernel: piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 Jun 18 12:09:17 fir-md1-s1 kernel: piix4_smbus 0000:00:14.0: Using register 0x2e for SMBus port selection Jun 18 12:09:17 fir-md1-s1 kernel: scsi 0:0:0:0: Attached scsi generic sg0 type 13 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:1:0: Attached scsi generic sg1 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: 3 command queues available Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: irq 238 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: irq 239 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 2 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 3 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 4 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 0 gets LSB 4 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 1 gets LSB 5 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 2 gets LSB 6 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:02:00.2: enabled Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: 5 command queues available Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: irq 241 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 0 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 1 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 2 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 3 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 4 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 0 gets LSB 1 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 1 gets LSB 2 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 2 gets LSB 3 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 3 gets LSB 4 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 4 gets LSB 5 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:03:00.1: enabled Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: 3 command queues available Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: irq 243 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: irq 244 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 2 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 3 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 4 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 0 gets LSB 4 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 1 gets LSB 5 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 2 gets LSB 6 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:41:00.2: enabled Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: 5 command queues available Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: irq 246 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 0 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 1 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 2 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 3 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 4 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 0 gets LSB 1 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 1 gets LSB 2 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 2 gets LSB 3 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 3 gets LSB 4 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 4 gets LSB 5 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:42:00.1: enabled Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: 3 command queues available Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: irq 248 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: irq 249 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 2 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 3 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 4 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 0 gets LSB 4 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 1 gets LSB 5 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 2 gets LSB 6 Jun 18 12:09:17 fir-md1-s1 kernel: ipmi device interface Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:85:00.2: enabled Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: 5 command queues available Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: irq 251 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 0 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 1 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 2 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 3 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 4 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 0 gets LSB 1 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 1 gets LSB 2 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 2 gets LSB 3 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 3 gets LSB 4 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 4 gets LSB 5 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:1:1: Attached scsi generic sg2 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:1:2: Attached scsi generic sg3 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: scsi 0:0:1:31: Attached scsi generic sg4 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:2:0: Attached scsi generic sg5 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:2:1: Attached scsi generic sg6 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: scsi 0:0:2:31: Attached scsi generic sg7 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:3:0: Attached scsi generic sg8 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:3:1: Attached scsi generic sg9 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:3:2: Attached scsi generic sg10 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: scsi 0:0:3:31: Attached scsi generic sg11 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:4:0: Attached scsi generic sg12 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 0:0:4:1: Attached scsi generic sg13 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: scsi 0:0:4:31: Attached scsi generic sg14 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: sd 1:2:0:0: Attached scsi generic sg15 type 0 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:86:00.1: enabled Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: 3 command queues available Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: irq 253 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: irq 254 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 2 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 3 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 4 can access 4 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 0 gets LSB 4 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 1 gets LSB 5 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 2 gets LSB 6 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c2:00.2: enabled Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: 5 command queues available Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: irq 256 for MSI/MSI-X Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 0 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 1 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 2 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 3 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 4 can access 7 LSB regions Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 0 gets LSB 1 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 1 gets LSB 2 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 2 gets LSB 3 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 3 gets LSB 4 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 4 gets LSB 5 Jun 18 12:09:17 fir-md1-s1 kernel: ccp 0000:c3:00.1: enabled Jun 18 12:09:17 fir-md1-s1 kernel: device-mapper: uevent: version 1.0.3 Jun 18 12:09:17 fir-md1-s1 kernel: IPMI System Interface driver. Jun 18 12:09:17 fir-md1-s1 kernel: input: PC Speaker as /devices/platform/pcspkr/input/input2 Jun 18 12:09:17 fir-md1-s1 kernel: device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com Jun 18 12:09:17 fir-md1-s1 kernel: ipmi_si ipmi_si.0: ipmi_platform: probing via SMBIOS Jun 18 12:09:17 fir-md1-s1 kernel: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 Jun 18 12:09:17 fir-md1-s1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine Jun 18 12:09:17 fir-md1-s1 kernel: ipmi_si IPI0001:00: ipmi_platform: probing via ACPI Jun 18 12:09:17 fir-md1-s1 kernel: ipmi_si IPI0001:00: [io 0x0ca8] regsize 1 spacing 4 irq 10 Jun 18 12:09:18 fir-md1-s1 kernel: mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) Jun 18 12:09:18 fir-md1-s1 kernel: ipmi_si ipmi_si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI Jun 18 12:09:18 fir-md1-s1 kernel: ipmi_si: Adding ACPI-specified kcs state machine Jun 18 12:09:18 fir-md1-s1 kernel: ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:1:0: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: cryptd: max_cpu_qlen set to 1000 Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:1:1: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:1:2: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: ipmi_si IPI0001:00: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. Jun 18 12:09:18 fir-md1-s1 kernel: ipmi_si IPI0001:00: Using irq 10 Jun 18 12:09:18 fir-md1-s1 kernel: ipmi_si IPI0001:00: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) Jun 18 12:09:18 fir-md1-s1 kernel: AVX2 version of gcm_enc/dec engaged. Jun 18 12:09:18 fir-md1-s1 kernel: scsi 0:0:1:31: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:2:0: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:2:1: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: AES CTR mode by8 optimization enabled Jun 18 12:09:18 fir-md1-s1 kernel: scsi 0:0:2:31: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: ipmi_si IPI0001:00: IPMI kcs interface initialized Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:3:0: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) Jun 18 12:09:18 fir-md1-s1 kernel: alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:3:1: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:3:2: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: kvm: Nested Paging enabled Jun 18 12:09:18 fir-md1-s1 kernel: scsi 0:0:3:31: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:4:0: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: sd 0:0:4:1: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: scsi 0:0:4:31: Embedded Enclosure Device Jun 18 12:09:18 fir-md1-s1 kernel: ses 0:0:0:0: Attached Enclosure device Jun 18 12:09:18 fir-md1-s1 kernel: MCE: In-kernel MCE decoding enabled. Jun 18 12:09:18 fir-md1-s1 kernel: AMD64 EDAC driver v3.4.0 Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 0). Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 2: 32767MB 3: 32767MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 2: 32767MB 3: 32767MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC0: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:18.3 Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 1). Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 2: 32767MB 3: 32767MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 2: 32767MB 3: 32767MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC1: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:19.3 Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 2). Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 2: 32767MB 3: 32767MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 2: 32767MB 3: 32767MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Jun 18 12:09:18 fir-md1-s1 kernel: EDAC MC2: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1a.3 Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Jun 18 12:09:18 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 3). Jun 18 12:09:19 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MC: 2: 32767MB 3: 32767MB Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Jun 18 12:09:19 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MC: 2: 32767MB 3: 32767MB Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Jun 18 12:09:19 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Jun 18 12:09:19 fir-md1-s1 kernel: EDAC MC3: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1b.3 Jun 18 12:09:19 fir-md1-s1 kernel: EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.0' (POLLED) Jun 18 12:09:20 fir-md1-s1 kernel: dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.3) Jun 18 12:09:36 fir-md1-s1 kernel: device-mapper: multipath round-robin: version 1.2.0 loaded Jun 18 12:10:03 fir-md1-s1 kernel: Adding 4194300k swap on /dev/sdk3. Priority:-2 extents:1 across:4194300k FS Jun 18 12:10:04 fir-md1-s1 kernel: FAT-fs (sdk1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck. Jun 18 12:10:04 fir-md1-s1 kernel: type=1305 audit(1560885004.098:3): audit_pid=17942 old=0 auid=4294967295 ses=4294967295 res=1 Jun 18 12:10:04 fir-md1-s1 kernel: RPC: Registered named UNIX socket transport module. Jun 18 12:10:04 fir-md1-s1 kernel: RPC: Registered udp transport module. Jun 18 12:10:04 fir-md1-s1 kernel: RPC: Registered tcp transport module. Jun 18 12:10:04 fir-md1-s1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Jun 18 12:10:04 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:04 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:04 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:04 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:04 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:04 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:04 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:04 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: slow_pci_heuristic:5202:(pid 18275): Max link speed = 100000, PCI BW = 126016 Jun 18 12:10:04 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(64) RxCqeCmprss(0) Jun 18 12:10:04 fir-md1-s1 kernel: mlx5_core 0000:84:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(64) RxCqeCmprss(0) Jun 18 12:10:05 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:05 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:05 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:05 fir-md1-s1 kernel: Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11 Jun 18 12:10:05 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 257 for MSI/MSI-X Jun 18 12:10:05 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 258 for MSI/MSI-X Jun 18 12:10:05 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 259 for MSI/MSI-X Jun 18 12:10:05 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 260 for MSI/MSI-X Jun 18 12:10:05 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 261 for MSI/MSI-X Jun 18 12:10:05 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready Jun 18 12:10:09 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: Link is up at 1000 Mbps, full duplex Jun 18 12:10:09 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: Flow control is on for TX and on for RX Jun 18 12:10:09 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: EEE is enabled Jun 18 12:10:09 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready Jun 18 12:10:09 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready Jun 18 12:10:09 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Jun 18 12:10:14 fir-md1-s1 kernel: FS-Cache: Loaded Jun 18 12:10:14 fir-md1-s1 kernel: FS-Cache: Netfs 'nfs' registered for caching Jun 18 12:10:14 fir-md1-s1 kernel: Key type dns_resolver registered Jun 18 12:10:14 fir-md1-s1 kernel: NFS: Registering the id_resolver key type Jun 18 12:10:14 fir-md1-s1 kernel: Key type id_resolver registered Jun 18 12:10:14 fir-md1-s1 kernel: Key type id_legacy registered Jun 18 12:10:46 fir-md1-s1 kernel: LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 Jun 18 12:10:46 fir-md1-s1 kernel: alg: No test for adler32 (adler32-zlib) Jun 18 12:10:47 fir-md1-s1 kernel: Lustre: Lustre: Build Version: 2.12.0_10_g4f75199 Jun 18 12:10:47 fir-md1-s1 kernel: LNet: Using FastReg for registration Jun 18 12:10:47 fir-md1-s1 kernel: LNetError: 7269:0:(o2iblnd_cb.c:2469:kiblnd_passive_connect()) Can't accept conn from 10.0.10.201@o2ib7 on NA (ib0:0:10.0.10.51): bad dst nid 10.0.10.51@o2ib7 Jun 18 12:10:47 fir-md1-s1 kernel: LNet: Added LNI 10.0.10.51@o2ib7 [8/256/0/180] Jun 18 12:11:18 fir-md1-s1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Jun 18 12:12:00 fir-md1-s1 kernel: LDISKFS-fs (dm-0): recovery complete Jun 18 12:12:00 fir-md1-s1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Jun 18 12:12:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to fbd75c70-2700-1de1-4de7-0793c5782012 (at 0@lo) Jun 18 12:12:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c579ffa9-959a-5f2e-006d-9d0dfdb5fa5a (at 10.8.17.26@o2ib6) Jun 18 12:12:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7888454b-080b-6943-cf4c-416d31bde0ec (at 10.9.104.28@o2ib4) Jun 18 12:12:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to e4594a87-2fe5-1bf8-dbe3-26a702178742 (at 10.8.0.67@o2ib6) Jun 18 12:12:09 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 18 12:12:14 fir-md1-s1 kernel: LDISKFS-fs warning (device dm-3): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Jun 18 12:12:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 323e9462-2806-288b-427b-09b4875db405 (at 10.0.10.52@o2ib7) Jun 18 12:12:16 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jun 18 12:12:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 14f72e02-ef06-defc-fe30-356a14ef5fda (at 10.9.109.29@o2ib4) Jun 18 12:12:25 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 18 12:12:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 6fb1a9aa-6234-c00b-63b2-a1a72639773f (at 10.8.7.19@o2ib6) Jun 18 12:12:42 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jun 18 12:12:56 fir-md1-s1 kernel: LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 Jun 18 12:12:57 fir-md1-s1 kernel: LDISKFS-fs (dm-3): recovery complete Jun 18 12:12:57 fir-md1-s1 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Jun 18 12:12:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.2.34@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 18 12:12:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.2.31@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 18 12:12:58 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jun 18 12:12:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.104.34@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 18 12:12:59 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 18 12:13:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.67@o2ib6, removing former export from same NID Jun 18 12:13:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.8.37@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 18 12:13:02 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jun 18 12:13:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.30.7@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 18 12:13:07 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jun 18 12:13:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.113.9@o2ib4, removing former export from same NID Jun 18 12:13:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 8990d314-6074-cf8b-1427-2287d94d8719 (at 10.8.23.29@o2ib6) Jun 18 12:13:14 fir-md1-s1 kernel: Lustre: Skipped 792 previous similar messages Jun 18 12:13:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.109.29@o2ib4, removing former export from same NID Jun 18 12:13:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 18 12:13:15 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jun 18 12:13:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Not available for connect from 10.8.0.67@o2ib6 (not set up) Jun 18 12:13:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Not available for connect from 10.9.108.60@o2ib4 (not set up) Jun 18 12:13:25 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 18 12:13:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Not available for connect from 10.9.101.58@o2ib4 (not set up) Jun 18 12:13:26 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 18 12:13:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Imperative Recovery not enabled, recovery window 300-900 Jun 18 12:13:27 fir-md1-s1 kernel: Lustre: fir-MDD0002: changelog on Jun 18 12:13:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Will be in recovery for at least 5:00, or until 1400 clients reconnect Jun 18 12:13:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.113.10@o2ib4, removing former export from same NID Jun 18 12:13:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.101.1@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 18 12:13:31 fir-md1-s1 kernel: LustreError: Skipped 505 previous similar messages Jun 18 12:13:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.104.14@o2ib4, removing former export from same NID Jun 18 12:13:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.104.28@o2ib4, removing former export from same NID Jun 18 12:13:43 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 18 12:14:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.7.19@o2ib6, removing former export from same NID Jun 18 12:14:04 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jun 18 12:14:05 fir-md1-s1 kernel: LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 Jun 18 12:14:05 fir-md1-s1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Jun 18 12:14:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.105.33@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 18 12:14:05 fir-md1-s1 kernel: LustreError: Skipped 939 previous similar messages Jun 18 12:14:06 fir-md1-s1 kernel: LustreError: 11-0: fir-MDT0002-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 Jun 18 12:14:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery not enabled, recovery window 300-900 Jun 18 12:14:06 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Jun 18 12:14:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Will be in recovery for at least 5:00, or until 1399 clients reconnect Jun 18 12:14:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Denying connection for new client 1e1769d3-ffba-a4ec-e5e5-cf0cf094a85d(at 10.8.8.37@o2ib6), waiting for 1400 known clients (3 recovered, 1352 in progress, and 0 evicted) already passed deadline 0:49 Jun 18 12:14:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 62d964d5-e241-336a-a44f-d2f1a33459f3 (at 10.9.105.41@o2ib4) Jun 18 12:14:18 fir-md1-s1 kernel: Lustre: Skipped 2049 previous similar messages Jun 18 12:14:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jun 18 12:14:38 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jun 18 12:14:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 1:15, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:14:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 1:16, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:14:44 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 18 12:14:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 1:17, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:14:45 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jun 18 12:14:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 1:19, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:14:47 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jun 18 12:14:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 1:23, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:14:51 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jun 18 12:15:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 1:32, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:15:00 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jun 18 12:15:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 1:48, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:15:16 fir-md1-s1 kernel: Lustre: Skipped 166 previous similar messages Jun 18 12:15:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Denying connection for new client 1e1769d3-ffba-a4ec-e5e5-cf0cf094a85d(at 10.8.8.37@o2ib6), waiting for 1400 known clients (4 recovered, 1395 in progress, and 0 evicted) already passed deadline 2:04 Jun 18 12:15:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.7.19@o2ib6, removing former export from same NID Jun 18 12:15:44 fir-md1-s1 kernel: Lustre: Skipped 1374 previous similar messages Jun 18 12:15:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 2:21, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:15:49 fir-md1-s1 kernel: Lustre: Skipped 1094 previous similar messages Jun 18 12:15:52 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client c579ffa9-959a-5f2e-006d-9d0dfdb5fa5a (at 10.8.17.26@o2ib6) in 229 seconds. I think it's dead, and I am evicting it. exp ffff8f453ecb7000, cur 1560885352 expire 1560885202 last 1560885123 Jun 18 12:16:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to aeb5fb0d-c687-a142-6a18-62fe99255a89 (at 10.8.30.8@o2ib6) Jun 18 12:16:26 fir-md1-s1 kernel: Lustre: Skipped 5929 previous similar messages Jun 18 12:16:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Denying connection for new client 1e1769d3-ffba-a4ec-e5e5-cf0cf094a85d(at 10.8.8.37@o2ib6), waiting for 1400 known clients (4 recovered, 1395 in progress, and 0 evicted) already passed deadline 3:20 Jun 18 12:16:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery already passed deadline 3:25, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:16:53 fir-md1-s1 kernel: Lustre: Skipped 1632 previous similar messages Jun 18 12:17:11 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 00a27512-1ff4-ce80-3c1f-4cfb4021ea64 (at 10.8.31.9@o2ib6) in 229 seconds. I think it's dead, and I am evicting it. exp ffff8f14e8c40400, cur 1560885431 expire 1560885281 last 1560885202 Jun 18 12:17:11 fir-md1-s1 kernel: Lustre: Skipped 1312 previous similar messages Jun 18 12:17:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.106.26@o2ib4, removing former export from same NID Jun 18 12:17:54 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jun 18 12:18:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: recovery is timed out, evict stale exports Jun 18 12:18:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: disconnecting 1 stale clients Jun 18 12:18:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Denying connection for new client 1e1769d3-ffba-a4ec-e5e5-cf0cf094a85d(at 10.8.8.37@o2ib6), waiting for 1400 known clients (4 recovered, 1395 in progress, and 1 evicted) already passed deadline 4:00 Jun 18 12:19:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery already passed deadline 4:48, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Jun 18 12:19:01 fir-md1-s1 kernel: Lustre: Skipped 4511 previous similar messages Jun 18 12:19:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: recovery is timed out, evict stale exports Jun 18 12:19:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: disconnecting 1396 stale clients Jun 18 12:19:13 fir-md1-s1 kernel: LustreError: 20943:0:(tgt_grant.c:248:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 94371840 Jun 18 12:19:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery over after 5:01, of 1400 clients 4 recovered and 1396 were evicted. Jun 18 12:19:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: recovery is timed out, evict stale exports Jun 18 12:19:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: disconnecting 1395 stale clients Jun 18 12:19:28 fir-md1-s1 kernel: LustreError: 20712:0:(tgt_grant.c:248:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 89374720 Jun 18 12:19:28 fir-md1-s1 kernel: LustreError: 20712:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 43 previous similar messages Jun 18 12:19:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Recovery over after 6:01, of 1400 clients 4 recovered and 1396 were evicted. Jun 18 12:19:45 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) in 232 seconds. I think it's dead, and I am evicting it. exp ffff8f150adbd000, cur 1560885585 expire 1560885435 last 1560885353 Jun 18 12:19:45 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jun 18 12:20:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 33d1aa9e-637b-a4b6-149b-4554121b9703 (at 10.9.109.56@o2ib4) reconnecting Jun 18 12:20:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 825512d6-7433-1c74-485b-b1a59d9ea8c8 (at 10.8.8.34@o2ib6) reconnecting Jun 18 12:20:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 564d73ec-5593-7fd1-5465-b4305978ee16 (at 10.8.17.8@o2ib6) reconnecting Jun 18 12:20:21 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 18 12:20:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client c24b8bff-f99c-4849-767d-bb11ab7dd32c (at 10.9.104.34@o2ib4) reconnecting Jun 18 12:20:24 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 18 12:20:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 2cc0bc1b-7a1f-9dab-b36c-c6206a02385d (at 10.8.20.20@o2ib6) reconnecting Jun 18 12:20:28 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 18 12:20:34 fir-md1-s1 kernel: LustreError: 21498:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 49152 Jun 18 12:20:34 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 118784 Jun 18 12:20:34 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 18 12:20:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 9dfc2bda-cf66-13a5-c506-30cd55e4267b (at 10.9.108.17@o2ib4) reconnecting Jun 18 12:20:37 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 18 12:20:39 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 18 12:20:39 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 26 previous similar messages Jun 18 12:20:41 fir-md1-s1 kernel: LustreError: 20510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 18 12:20:41 fir-md1-s1 kernel: LustreError: 20510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 33 previous similar messages Jun 18 12:20:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 92d3fcc8-c103-27a1-5dc1-c88d1de34211 (at 10.8.12.30@o2ib6) Jun 18 12:20:42 fir-md1-s1 kernel: Lustre: Skipped 12151 previous similar messages Jun 18 12:20:49 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 18 12:20:49 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 263 previous similar messages Jun 18 12:20:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jun 18 12:20:55 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jun 18 12:20:59 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 12:20:59 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 70 previous similar messages Jun 18 12:21:03 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 8257ad81-12d5-f269-3c44-478c2a180d99 (at 10.8.17.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14b3de2000, cur 1560885663 expire 1560885513 last 1560885436 Jun 18 12:21:03 fir-md1-s1 kernel: Lustre: Skipped 1302 previous similar messages Jun 18 12:21:15 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 18 12:21:15 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 32 previous similar messages Jun 18 12:21:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 2f3fe604-ebdc-987d-cf70-34fded524b5d (at 10.8.21.23@o2ib6) reconnecting Jun 18 12:21:27 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jun 18 12:21:49 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 18 12:21:49 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 53 previous similar messages Jun 18 12:22:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.18.18@o2ib6, removing former export from same NID Jun 18 12:22:11 fir-md1-s1 kernel: Lustre: Skipped 1681 previous similar messages Jun 18 12:22:54 fir-md1-s1 kernel: LustreError: 21538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 12:22:54 fir-md1-s1 kernel: LustreError: 21538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 257 previous similar messages Jun 18 12:25:05 fir-md1-s1 kernel: LustreError: 21541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 18 12:25:05 fir-md1-s1 kernel: LustreError: 21541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 223 previous similar messages Jun 18 12:29:29 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 12:29:29 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 230 previous similar messages Jun 18 12:30:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 18 12:30:01 fir-md1-s1 kernel: Lustre: Skipped 630 previous similar messages Jun 18 12:38:02 fir-md1-s1 kernel: LustreError: 21743:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 18 12:38:02 fir-md1-s1 kernel: LustreError: 21743:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 857 previous similar messages Jun 18 12:48:03 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 12:48:03 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1173 previous similar messages Jun 18 12:58:05 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 12:58:05 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 496 previous similar messages Jun 18 13:08:05 fir-md1-s1 kernel: LustreError: 21684:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 13:08:05 fir-md1-s1 kernel: LustreError: 21684:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 254 previous similar messages Jun 18 13:08:56 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 13:18:16 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 13:18:16 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 524 previous similar messages Jun 18 13:28:17 fir-md1-s1 kernel: LustreError: 21793:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 13:28:17 fir-md1-s1 kernel: LustreError: 21793:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 18 13:38:21 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 18 13:38:21 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 484 previous similar messages Jun 18 13:48:26 fir-md1-s1 kernel: LustreError: 21742:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 18 13:48:26 fir-md1-s1 kernel: LustreError: 21742:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 384 previous similar messages Jun 18 13:52:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) reconnecting Jun 18 13:52:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 18 13:52:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 18 13:52:46 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jun 18 13:58:29 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 13:58:29 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 388 previous similar messages Jun 18 14:07:21 fir-md1-s1 kernel: perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 Jun 18 14:08:30 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 14:08:30 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 320 previous similar messages Jun 18 14:19:38 fir-md1-s1 kernel: LustreError: 20510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 18 14:19:38 fir-md1-s1 kernel: LustreError: 20510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 222 previous similar messages Jun 18 14:29:44 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 14:29:44 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 191 previous similar messages Jun 18 14:35:28 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 14:39:55 fir-md1-s1 kernel: LustreError: 21566:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 18 14:39:55 fir-md1-s1 kernel: LustreError: 21566:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 283 previous similar messages Jun 18 14:49:55 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 18 14:49:55 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 205 previous similar messages Jun 18 15:00:06 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 15:00:06 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 216 previous similar messages Jun 18 15:10:07 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 15:10:07 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 294 previous similar messages Jun 18 15:18:25 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 15:20:07 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 15:20:07 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 18 15:22:44 fir-md1-s1 kernel: perf: interrupt took too long (3155 > 3126), lowering kernel.perf_event_max_sample_rate to 63000 Jun 18 15:30:15 fir-md1-s1 kernel: LustreError: 21792:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 15:30:15 fir-md1-s1 kernel: LustreError: 21792:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 455 previous similar messages Jun 18 15:33:04 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 15:37:41 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 15:40:18 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 15:40:18 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 388 previous similar messages Jun 18 15:43:59 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 15:43:59 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 18 15:46:02 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 15:50:41 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 18 15:50:41 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 421 previous similar messages Jun 18 16:00:44 fir-md1-s1 kernel: LustreError: 21794:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 18 16:00:44 fir-md1-s1 kernel: LustreError: 21794:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 332 previous similar messages Jun 18 16:01:14 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:01:14 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages Jun 18 16:05:43 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:10:50 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 16:10:50 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 462 previous similar messages Jun 18 16:20:52 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 16:20:52 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 379 previous similar messages Jun 18 16:28:42 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:30:54 fir-md1-s1 kernel: LustreError: 22973:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 16:30:54 fir-md1-s1 kernel: LustreError: 22973:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 407 previous similar messages Jun 18 16:34:28 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:35:17 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:38:37 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:40:00 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:40:57 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 16:40:57 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 358 previous similar messages Jun 18 16:43:37 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:43:37 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages Jun 18 16:46:33 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 16:50:59 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 18 16:50:59 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 375 previous similar messages Jun 18 16:51:37 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:01:01 fir-md1-s1 kernel: LustreError: 22973:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 17:01:01 fir-md1-s1 kernel: LustreError: 22973:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 285 previous similar messages Jun 18 17:04:42 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:05:23 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:05:23 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages Jun 18 17:07:27 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:07:27 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 18 17:08:25 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:09:38 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:09:38 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages Jun 18 17:11:05 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 18 17:11:05 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 252 previous similar messages Jun 18 17:19:55 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:19:55 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages Jun 18 17:21:22 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 17:21:22 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 244 previous similar messages Jun 18 17:28:53 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:28:53 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages Jun 18 17:29:10 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 18 17:29:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 545f12c1-4799-a254-b9c4-f75f43e1bc5b (at 10.8.27.23@o2ib6) reconnecting Jun 18 17:29:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to f2f779cf-d459-667d-6b56-c14a76db50bb (at 10.8.27.23@o2ib6) Jun 18 17:29:17 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 18 17:29:35 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 18 17:29:35 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Jun 18 17:29:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 545f12c1-4799-a254-b9c4-f75f43e1bc5b (at 10.8.27.23@o2ib6) reconnecting Jun 18 17:29:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 29b52eb8-dab6-4b88-7a0d-057d59d63b47 (at 10.8.17.22@o2ib6) Jun 18 17:29:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 18 17:31:47 fir-md1-s1 kernel: LustreError: 21684:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 17:31:47 fir-md1-s1 kernel: LustreError: 21684:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 195 previous similar messages Jun 18 17:38:35 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:38:35 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages Jun 18 17:41:57 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 17:41:57 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 223 previous similar messages Jun 18 17:46:25 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 18 17:46:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 564d73ec-5593-7fd1-5465-b4305978ee16 (at 10.8.17.8@o2ib6) reconnecting Jun 18 17:46:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 5132afab-5b1d-c7e5-9316-17cfeee10d24 (at 10.8.17.8@o2ib6) Jun 18 17:46:56 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 18 17:48:41 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:48:41 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages Jun 18 17:51:57 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 18 17:51:57 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1009 previous similar messages Jun 18 17:59:30 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 17:59:30 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages Jun 18 18:02:07 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 18:02:07 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1677 previous similar messages Jun 18 18:10:18 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 18:10:18 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages Jun 18 18:12:09 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 18:12:09 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2381 previous similar messages Jun 18 18:21:54 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 18:21:54 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 19 previous similar messages Jun 18 18:22:10 fir-md1-s1 kernel: LustreError: 21684:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 18:22:10 fir-md1-s1 kernel: LustreError: 21684:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1508 previous similar messages Jun 18 18:31:56 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 18:31:56 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages Jun 18 18:32:10 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 18:32:10 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 819 previous similar messages Jun 18 18:42:14 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 18 18:42:14 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 974 previous similar messages Jun 18 18:42:38 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 18:42:38 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages Jun 18 18:52:14 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 18:52:14 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22541 previous similar messages Jun 18 18:53:10 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 18:53:10 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages Jun 18 19:02:16 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 18 19:02:16 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 39085 previous similar messages Jun 18 19:04:06 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 19:04:06 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 75 previous similar messages Jun 18 19:12:16 fir-md1-s1 kernel: LustreError: 27482:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 18 19:12:16 fir-md1-s1 kernel: LustreError: 27482:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1486 previous similar messages Jun 18 19:16:32 fir-md1-s1 kernel: Lustre: 21418:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 19:16:32 fir-md1-s1 kernel: Lustre: 21418:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages Jun 18 19:22:20 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 19:22:20 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 976 previous similar messages Jun 18 19:27:34 fir-md1-s1 kernel: Lustre: 20501:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1509cf8050 x1636444251328064/t0(0) o3->2ca8c1ab-ca57-7d24-398b-275ee2691945@10.9.112.16@o2ib4:9/0 lens 488/440 e 0 to 0 dl 1560911259 ref 2 fl Interpret:/0/0 rc 0/0 Jun 18 19:27:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 2ca8c1ab-ca57-7d24-398b-275ee2691945 (at 10.9.112.16@o2ib4) reconnecting Jun 18 19:27:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 286d4aef-dd39-033a-885a-1b2f68dad8ee (at 10.9.112.16@o2ib4) Jun 18 19:27:56 fir-md1-s1 kernel: LustreError: 20500:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8f1509cf8050 x1636444251328064/t0(0) o3->2ca8c1ab-ca57-7d24-398b-275ee2691945@10.9.112.16@o2ib4:9/0 lens 488/440 e 0 to 0 dl 1560911259 ref 1 fl Interpret:/0/0 rc 0/0 Jun 18 19:27:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 2ca8c1ab-ca57-7d24-398b-275ee2691945 (at 10.9.112.16@o2ib4), client will retry: rc -107 Jun 18 19:27:56 fir-md1-s1 kernel: Lustre: 20500:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:17s); client may timeout. req@ffff8f1509cf8050 x1636444251328064/t0(0) o3->2ca8c1ab-ca57-7d24-398b-275ee2691945@10.9.112.16@o2ib4:9/0 lens 488/440 e 0 to 0 dl 1560911259 ref 1 fl Complete:/0/ffffffff rc -107/-1 Jun 18 19:28:14 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 19:28:14 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 32 previous similar messages Jun 18 19:32:28 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 18 19:32:28 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1433 previous similar messages Jun 18 19:38:27 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 19:38:27 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 62 previous similar messages Jun 18 19:42:28 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 18 19:42:28 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1787 previous similar messages Jun 18 19:49:32 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 19:49:32 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 121 previous similar messages Jun 18 19:52:32 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 18 19:52:32 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 57799 previous similar messages Jun 18 20:00:22 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 20:00:22 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages Jun 18 20:03:16 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 20:03:16 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3013 previous similar messages Jun 18 20:10:56 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 20:10:56 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 27 previous similar messages Jun 18 20:13:18 fir-md1-s1 kernel: LustreError: 22058:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 20:13:18 fir-md1-s1 kernel: LustreError: 22058:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 456 previous similar messages Jun 18 20:23:30 fir-md1-s1 kernel: LustreError: 21794:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 20:23:30 fir-md1-s1 kernel: LustreError: 21794:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 682 previous similar messages Jun 18 20:24:00 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 20:24:00 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 49 previous similar messages Jun 18 20:33:30 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 18 20:33:30 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 995 previous similar messages Jun 18 20:35:16 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 20:35:16 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 23 previous similar messages Jun 18 20:43:34 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 18 20:43:34 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 863 previous similar messages Jun 18 20:46:28 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 20:46:28 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 62 previous similar messages Jun 18 20:53:44 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 18 20:53:44 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 430 previous similar messages Jun 18 20:57:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client a1402e9b-5e48-acd3-204d-e410e8c1eb0b (at 10.8.2.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f350405e800, cur 1560916663 expire 1560916513 last 1560916436 Jun 18 20:57:43 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jun 18 20:57:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client a1402e9b-5e48-acd3-204d-e410e8c1eb0b (at 10.8.2.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f350405f400, cur 1560916666 expire 1560916516 last 1560916439 Jun 18 20:57:46 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 18 20:58:25 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 20:58:25 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 80 previous similar messages Jun 18 21:03:45 fir-md1-s1 kernel: LustreError: 23107:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 21:03:45 fir-md1-s1 kernel: LustreError: 23107:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 529 previous similar messages Jun 18 21:08:51 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 21:08:51 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 93 previous similar messages Jun 18 21:13:45 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 18 21:13:45 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 330 previous similar messages Jun 18 21:18:52 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 21:18:52 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 111 previous similar messages Jun 18 21:23:46 fir-md1-s1 kernel: LustreError: 21792:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 21:23:46 fir-md1-s1 kernel: LustreError: 21792:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2183 previous similar messages Jun 18 21:31:54 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 21:31:54 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 41 previous similar messages Jun 18 21:33:46 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 18 21:33:46 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 563 previous similar messages Jun 18 21:43:50 fir-md1-s1 kernel: LustreError: 25972:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 18 21:43:50 fir-md1-s1 kernel: LustreError: 25972:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 775 previous similar messages Jun 18 21:44:42 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 21:44:42 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 105 previous similar messages Jun 18 21:53:54 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 18 21:53:54 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 810 previous similar messages Jun 18 21:55:16 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 21:55:16 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 67 previous similar messages Jun 18 22:03:55 fir-md1-s1 kernel: LustreError: 21545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 22:03:55 fir-md1-s1 kernel: LustreError: 21545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 43806 previous similar messages Jun 18 22:08:03 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 22:08:03 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 378 previous similar messages Jun 18 22:13:58 fir-md1-s1 kernel: LustreError: 22058:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 18 22:13:58 fir-md1-s1 kernel: LustreError: 22058:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15736 previous similar messages Jun 18 22:18:12 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 22:18:12 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 264 previous similar messages Jun 18 22:24:02 fir-md1-s1 kernel: LustreError: 21792:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 94208 GRANT, real grant 0 Jun 18 22:24:02 fir-md1-s1 kernel: LustreError: 21792:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 310 previous similar messages Jun 18 22:28:15 fir-md1-s1 kernel: Lustre: 20457:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 22:28:15 fir-md1-s1 kernel: Lustre: 20457:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 138 previous similar messages Jun 18 22:34:26 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 22:34:26 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 314 previous similar messages Jun 18 22:38:16 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 22:38:16 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 284 previous similar messages Jun 18 22:44:50 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 22:44:50 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 221 previous similar messages Jun 18 22:48:48 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 22:48:48 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 146 previous similar messages Jun 18 22:54:56 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 22:54:56 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 295 previous similar messages Jun 18 22:58:51 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 22:58:51 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 219 previous similar messages Jun 18 23:05:07 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 18 23:05:07 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 336 previous similar messages Jun 18 23:13:24 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 23:13:24 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 142 previous similar messages Jun 18 23:15:58 fir-md1-s1 kernel: LustreError: 21684:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 23:15:58 fir-md1-s1 kernel: LustreError: 21684:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 220 previous similar messages Jun 18 23:26:01 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 23:26:01 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 203 previous similar messages Jun 18 23:26:16 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 23:26:16 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 111 previous similar messages Jun 18 23:36:03 fir-md1-s1 kernel: LustreError: 25972:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 18 23:36:03 fir-md1-s1 kernel: LustreError: 25972:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 265 previous similar messages Jun 18 23:37:49 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 23:37:49 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 81 previous similar messages Jun 18 23:46:07 fir-md1-s1 kernel: LustreError: 20507:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 18 23:46:07 fir-md1-s1 kernel: LustreError: 20507:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 244 previous similar messages Jun 18 23:48:55 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 18 23:48:55 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 148 previous similar messages Jun 18 23:56:23 fir-md1-s1 kernel: LustreError: 21291:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 18 23:56:23 fir-md1-s1 kernel: LustreError: 21291:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 288 previous similar messages Jun 19 00:00:26 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 00:00:26 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 255 previous similar messages Jun 19 00:06:28 fir-md1-s1 kernel: LustreError: 21450:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 00:06:28 fir-md1-s1 kernel: LustreError: 21450:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 269 previous similar messages Jun 19 00:12:19 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 00:12:19 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 404 previous similar messages Jun 19 00:16:37 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 00:16:37 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 312 previous similar messages Jun 19 00:26:00 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 00:26:00 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 438 previous similar messages Jun 19 00:26:38 fir-md1-s1 kernel: LustreError: 21741:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 00:26:38 fir-md1-s1 kernel: LustreError: 21741:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 264 previous similar messages Jun 19 00:36:40 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 00:36:40 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 276 previous similar messages Jun 19 00:37:46 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 00:37:46 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 638 previous similar messages Jun 19 00:41:45 fir-md1-s1 kernel: perf: interrupt took too long (3949 > 3943), lowering kernel.perf_event_max_sample_rate to 50000 Jun 19 00:46:42 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 00:46:42 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 299 previous similar messages Jun 19 00:50:39 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 00:50:39 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 281 previous similar messages Jun 19 00:56:46 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 00:56:46 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 340 previous similar messages Jun 19 01:00:48 fir-md1-s1 kernel: Lustre: 21418:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 01:00:48 fir-md1-s1 kernel: Lustre: 21418:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 251 previous similar messages Jun 19 01:06:51 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 01:06:51 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1389 previous similar messages Jun 19 01:11:02 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 01:11:02 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 460 previous similar messages Jun 19 01:16:54 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 01:16:54 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 318 previous similar messages Jun 19 01:21:14 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 01:21:14 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 211 previous similar messages Jun 19 01:27:05 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 01:27:05 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 706 previous similar messages Jun 19 01:31:20 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 01:31:20 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 269 previous similar messages Jun 19 01:37:18 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 01:37:18 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 762 previous similar messages Jun 19 01:44:13 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 01:44:13 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 295 previous similar messages Jun 19 01:47:31 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 01:47:31 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 758 previous similar messages Jun 19 01:57:17 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 01:57:17 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 179 previous similar messages Jun 19 01:57:38 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 01:57:38 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 735 previous similar messages Jun 19 02:07:40 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 02:07:40 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 723 previous similar messages Jun 19 02:10:21 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 02:10:21 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 272 previous similar messages Jun 19 02:17:50 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 02:17:50 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 790 previous similar messages Jun 19 02:23:04 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 02:23:04 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 235 previous similar messages Jun 19 02:27:53 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 02:27:53 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1832 previous similar messages Jun 19 02:35:38 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 02:35:38 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 553 previous similar messages Jun 19 02:37:53 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 02:37:53 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 720 previous similar messages Jun 19 02:46:59 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 02:46:59 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 399 previous similar messages Jun 19 02:48:04 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 122880 GRANT, real grant 0 Jun 19 02:48:04 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 595 previous similar messages Jun 19 02:58:10 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 65536 GRANT, real grant 0 Jun 19 02:58:10 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 181 previous similar messages Jun 19 02:58:20 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 02:58:20 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 297 previous similar messages Jun 19 03:08:20 fir-md1-s1 kernel: LustreError: 21742:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 19 03:08:20 fir-md1-s1 kernel: LustreError: 21742:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 334 previous similar messages Jun 19 03:08:31 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 03:08:31 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 168 previous similar messages Jun 19 03:18:22 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 03:18:22 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1471 previous similar messages Jun 19 03:18:40 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 03:18:40 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 168 previous similar messages Jun 19 03:28:27 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 03:28:27 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 464 previous similar messages Jun 19 03:28:42 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 03:28:42 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 163 previous similar messages Jun 19 03:38:32 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 03:38:32 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 718 previous similar messages Jun 19 03:39:02 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 03:39:02 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 231 previous similar messages Jun 19 03:48:36 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 03:48:36 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1046 previous similar messages Jun 19 03:49:04 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 03:49:04 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 714 previous similar messages Jun 19 03:58:37 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 94208 GRANT, real grant 0 Jun 19 03:58:37 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 929 previous similar messages Jun 19 03:59:08 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 03:59:08 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 593 previous similar messages Jun 19 04:08:38 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 04:08:38 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1779 previous similar messages Jun 19 04:09:26 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 04:09:26 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 282 previous similar messages Jun 19 04:18:44 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 04:18:44 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2222 previous similar messages Jun 19 04:19:32 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 04:19:32 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 635 previous similar messages Jun 19 04:28:56 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 04:28:56 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 776 previous similar messages Jun 19 04:29:38 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 04:29:38 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 565 previous similar messages Jun 19 04:38:57 fir-md1-s1 kernel: LustreError: 21713:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 04:38:57 fir-md1-s1 kernel: LustreError: 21713:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1076 previous similar messages Jun 19 04:39:43 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 04:39:43 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 157 previous similar messages Jun 19 04:49:04 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 04:49:04 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1406 previous similar messages Jun 19 04:49:50 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 04:49:50 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 403 previous similar messages Jun 19 04:59:13 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 04:59:13 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1324 previous similar messages Jun 19 04:59:58 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 04:59:58 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 207 previous similar messages Jun 19 05:09:17 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 19 05:09:17 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1180 previous similar messages Jun 19 05:10:52 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 05:10:52 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 270 previous similar messages Jun 19 05:19:19 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 05:19:19 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 957 previous similar messages Jun 19 05:20:54 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 05:20:54 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 455 previous similar messages Jun 19 05:29:27 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 05:29:27 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1314 previous similar messages Jun 19 05:31:00 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 05:31:00 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 297 previous similar messages Jun 19 05:39:28 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 05:39:28 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1375 previous similar messages Jun 19 05:41:05 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 05:41:05 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 532 previous similar messages Jun 19 05:49:34 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 05:49:34 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1322 previous similar messages Jun 19 05:51:06 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 05:51:06 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 399 previous similar messages Jun 19 05:59:41 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 05:59:41 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1197 previous similar messages Jun 19 06:01:23 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 06:01:23 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 480 previous similar messages Jun 19 06:09:44 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 06:09:44 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1182 previous similar messages Jun 19 06:11:38 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 06:11:38 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 214 previous similar messages Jun 19 06:19:45 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 06:19:45 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 893 previous similar messages Jun 19 06:21:38 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 06:21:38 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 597 previous similar messages Jun 19 06:29:50 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 19 06:29:50 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 560 previous similar messages Jun 19 06:31:59 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 06:31:59 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 361 previous similar messages Jun 19 06:39:51 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 06:39:51 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 824 previous similar messages Jun 19 06:42:05 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 06:42:05 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 274 previous similar messages Jun 19 06:49:52 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 06:49:52 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 828 previous similar messages Jun 19 06:52:09 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 06:52:09 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 227 previous similar messages Jun 19 06:59:53 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 147456 GRANT, real grant 0 Jun 19 06:59:53 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1021 previous similar messages Jun 19 07:02:31 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 07:02:31 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 175 previous similar messages Jun 19 07:09:54 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 151552 GRANT, real grant 0 Jun 19 07:09:54 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 989 previous similar messages Jun 19 07:12:37 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 07:12:37 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 242 previous similar messages Jun 19 07:19:54 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 07:19:54 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1053 previous similar messages Jun 19 07:22:42 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 07:22:42 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 138 previous similar messages Jun 19 07:29:54 fir-md1-s1 kernel: LustreError: 21450:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 07:29:54 fir-md1-s1 kernel: LustreError: 21450:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1027 previous similar messages Jun 19 07:39:28 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 07:39:28 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 48 previous similar messages Jun 19 07:39:55 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 07:39:55 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 894 previous similar messages Jun 19 07:50:05 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 07:50:05 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 792 previous similar messages Jun 19 07:50:56 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 07:50:56 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 246 previous similar messages Jun 19 08:00:08 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 08:00:08 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 828 previous similar messages Jun 19 08:02:46 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 08:02:46 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 111 previous similar messages Jun 19 08:10:09 fir-md1-s1 kernel: LustreError: 25998:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 08:10:09 fir-md1-s1 kernel: LustreError: 25998:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1088 previous similar messages Jun 19 08:13:05 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 08:13:05 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 143 previous similar messages Jun 19 08:20:18 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 08:20:18 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1292 previous similar messages Jun 19 08:23:22 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 08:23:22 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 301 previous similar messages Jun 19 08:30:22 fir-md1-s1 kernel: LustreError: 23107:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 08:30:22 fir-md1-s1 kernel: LustreError: 23107:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 34339 previous similar messages Jun 19 08:33:57 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 08:33:57 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages Jun 19 08:40:22 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 122880 GRANT, real grant 0 Jun 19 08:40:22 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 26179 previous similar messages Jun 19 08:45:39 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 08:45:39 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 60 previous similar messages Jun 19 08:50:22 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 08:50:22 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1185 previous similar messages Jun 19 08:56:44 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 08:56:44 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 35 previous similar messages Jun 19 09:00:24 fir-md1-s1 kernel: LustreError: 23093:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 19 09:00:24 fir-md1-s1 kernel: LustreError: 23093:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 301 previous similar messages Jun 19 09:09:07 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 09:09:07 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 30 previous similar messages Jun 19 09:10:27 fir-md1-s1 kernel: LustreError: 22434:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 09:10:27 fir-md1-s1 kernel: LustreError: 22434:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 197 previous similar messages Jun 19 09:20:28 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 09:20:28 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 294 previous similar messages Jun 19 09:26:05 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 09:26:05 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 39 previous similar messages Jun 19 09:30:44 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 09:30:44 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 219 previous similar messages Jun 19 09:38:09 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 09:38:09 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages Jun 19 09:40:51 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 09:40:51 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 256 previous similar messages Jun 19 09:48:24 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 09:48:24 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 49 previous similar messages Jun 19 09:51:04 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 09:51:04 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 286 previous similar messages Jun 19 10:01:07 fir-md1-s1 kernel: LustreError: 22975:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 10:01:07 fir-md1-s1 kernel: LustreError: 22975:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2282 previous similar messages Jun 19 10:06:06 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 10:06:06 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages Jun 19 10:11:10 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 19 10:11:10 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 475 previous similar messages Jun 19 10:16:14 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 10:16:14 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 161 previous similar messages Jun 19 10:21:14 fir-md1-s1 kernel: LustreError: 21742:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 19 10:21:14 fir-md1-s1 kernel: LustreError: 21742:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 515 previous similar messages Jun 19 10:26:18 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 10:26:18 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages Jun 19 10:31:14 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 10:31:14 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 567 previous similar messages Jun 19 10:38:19 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 10:38:19 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages Jun 19 10:41:17 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 10:41:17 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 482 previous similar messages Jun 19 10:51:20 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 10:51:20 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 557 previous similar messages Jun 19 10:55:45 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 10:55:45 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages Jun 19 11:01:20 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 11:01:20 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 512 previous similar messages Jun 19 11:08:56 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 11:08:56 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 58 previous similar messages Jun 19 11:11:28 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 19 11:11:28 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 571 previous similar messages Jun 19 11:21:28 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 11:21:28 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 711 previous similar messages Jun 19 11:21:45 fir-md1-s1 kernel: Lustre: 21418:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 11:21:45 fir-md1-s1 kernel: Lustre: 21418:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages Jun 19 11:31:30 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 19 11:31:30 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 814 previous similar messages Jun 19 11:31:47 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 11:31:47 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 44 previous similar messages Jun 19 11:41:35 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 11:41:35 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 556 previous similar messages Jun 19 11:43:12 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 11:43:12 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages Jun 19 11:51:39 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 11:51:39 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 388 previous similar messages Jun 19 11:56:27 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 11:56:27 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages Jun 19 12:01:54 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 12:01:54 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 400 previous similar messages Jun 19 12:11:57 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 45056 GRANT, real grant 0 Jun 19 12:11:57 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 393 previous similar messages Jun 19 12:22:04 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 12:22:04 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jun 19 12:32:07 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 12:32:07 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 308 previous similar messages Jun 19 12:42:07 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 12:42:07 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 288 previous similar messages Jun 19 12:47:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 69994cc7-6cad-e493-9816-76214dd8e291 (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2502e19800, cur 1560973672 expire 1560973522 last 1560973445 Jun 19 12:52:07 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 12:52:07 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 329 previous similar messages Jun 19 13:02:09 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 19 13:02:09 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 759 previous similar messages Jun 19 13:12:09 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 13:12:09 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1401 previous similar messages Jun 19 13:22:12 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 13:22:12 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1397 previous similar messages Jun 19 13:32:48 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 32768 GRANT, real grant 0 Jun 19 13:32:48 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1165 previous similar messages Jun 19 13:42:48 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 19 13:42:48 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 573 previous similar messages Jun 19 13:52:55 fir-md1-s1 kernel: LustreError: 20510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 13:52:55 fir-md1-s1 kernel: LustreError: 20510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1035 previous similar messages Jun 19 14:02:57 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 19 14:02:57 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 124 previous similar messages Jun 19 14:12:58 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 65536 GRANT, real grant 0 Jun 19 14:12:58 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 516 previous similar messages Jun 19 14:23:01 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 14:23:01 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 411 previous similar messages Jun 19 14:34:43 fir-md1-s1 kernel: LustreError: 21740:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 19 14:34:43 fir-md1-s1 kernel: LustreError: 21740:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 673 previous similar messages Jun 19 14:44:48 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 14:44:48 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 407 previous similar messages Jun 19 14:54:51 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 14:54:51 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 514 previous similar messages Jun 19 15:04:51 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 15:04:51 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 752 previous similar messages Jun 19 15:15:37 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 19 15:15:37 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1957 previous similar messages Jun 19 15:26:01 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 15:26:01 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 416 previous similar messages Jun 19 15:36:17 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 19 15:36:17 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 545 previous similar messages Jun 19 15:46:21 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 110592 GRANT, real grant 0 Jun 19 15:46:21 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16424 previous similar messages Jun 19 16:03:49 fir-md1-s1 kernel: LustreError: 21566:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 19 16:03:49 fir-md1-s1 kernel: LustreError: 21566:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 42796 previous similar messages Jun 19 17:03:09 fir-md1-s1 kernel: sd 0:0:3:1: Inquiry data has changed Jun 19 17:03:21 fir-md1-s1 kernel: sd 0:0:3:1: Inquiry data has changed Jun 19 17:03:28 fir-md1-s1 kernel: sd 0:0:1:0: Inquiry data has changed Jun 19 17:03:38 fir-md1-s1 kernel: sd 0:0:1:0: Inquiry data has changed Jun 19 17:10:04 fir-md1-s1 kernel: sd 0:0:1:1: Inquiry data has changed Jun 19 17:10:04 fir-md1-s1 kernel: sd 0:0:1:2: Inquiry data has changed Jun 19 17:10:04 fir-md1-s1 kernel: sd 0:0:3:0: Inquiry data has changed Jun 19 17:10:04 fir-md1-s1 kernel: sd 0:0:3:2: Inquiry data has changed Jun 19 18:44:49 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 18:44:49 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages Jun 19 20:46:00 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 20:47:02 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 19 20:47:02 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages Jun 19 22:32:13 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 19 22:32:13 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7 previous similar messages Jun 19 22:33:56 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 19 22:33:56 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 19 22:37:21 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 19 22:57:44 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 19 22:57:44 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jun 19 23:15:19 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 19 23:15:19 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 19 23:15:26 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 19 23:16:22 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 19 23:49:12 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 00:57:10 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 00:57:10 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 20 00:57:16 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 00:57:16 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jun 20 00:57:22 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 00:57:39 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 00:57:39 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 55 previous similar messages Jun 20 00:58:01 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 110592 GRANT, real grant 0 Jun 20 00:58:01 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 797 previous similar messages Jun 20 00:58:53 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 20 00:58:53 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 165 previous similar messages Jun 20 01:01:26 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 61440 GRANT, real grant 0 Jun 20 01:01:26 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 20 01:16:09 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 01:16:37 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 01:16:37 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jun 20 01:17:19 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 57344 GRANT, real grant 0 Jun 20 01:17:19 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 57 previous similar messages Jun 20 01:18:39 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 20 01:18:39 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 107 previous similar messages Jun 20 01:21:12 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 01:21:12 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 78 previous similar messages Jun 20 01:26:17 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 01:26:17 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 84 previous similar messages Jun 20 01:36:23 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 01:36:23 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 489 previous similar messages Jun 20 01:47:08 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 01:47:08 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 443 previous similar messages Jun 20 01:57:14 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 01:57:14 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 473 previous similar messages Jun 20 02:07:14 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 02:07:14 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 451 previous similar messages Jun 20 02:17:18 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 02:17:18 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 441 previous similar messages Jun 20 02:27:53 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 02:27:53 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1629 previous similar messages Jun 20 02:37:58 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jun 20 02:37:58 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 498 previous similar messages Jun 20 02:47:59 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 02:47:59 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 399 previous similar messages Jun 20 02:58:04 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 02:58:04 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 46 previous similar messages Jun 20 03:08:24 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 03:08:24 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jun 20 03:18:26 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 03:18:26 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1300 previous similar messages Jun 20 03:22:00 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026113/real 1561026113] req@ffff8f1c772f4800 x1636708563212800/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026120 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 20 03:22:07 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026120/real 1561026120] req@ffff8f1c772f4800 x1636708563212800/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026127 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 20 03:22:07 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 20 03:22:08 fir-md1-s1 kernel: Lustre: 21482:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1e057f0000 x1636580163248912/t0(0) o101->804bb2d0-a656-6c01-b0db-5b53058fb0f9@10.8.9.9@o2ib6:13/0 lens 480/568 e 1 to 0 dl 1561026133 ref 2 fl Interpret:/0/0 rc 0/0 Jun 20 03:22:14 fir-md1-s1 kernel: Lustre: 20460:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026127/real 1561026127] req@ffff8f24c36c2700 x1636708563213008/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026134 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 20 03:22:14 fir-md1-s1 kernel: Lustre: 20460:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 20 03:22:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 804bb2d0-a656-6c01-b0db-5b53058fb0f9 (at 10.8.9.9@o2ib6) reconnecting Jun 20 03:22:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 20 03:22:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 00a6bf4a-1a11-675b-07eb-2392e93c70c7 (at 10.8.29.8@o2ib6) reconnecting Jun 20 03:22:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 220a94f1-3873-c0d2-13c3-2a8b3b58132e (at 10.8.29.8@o2ib6) Jun 20 03:22:21 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026134/real 1561026134] req@ffff8f1c772f4800 x1636708563212800/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026141 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 20 03:22:21 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 20 03:22:28 fir-md1-s1 kernel: Lustre: 20460:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026141/real 1561026141] req@ffff8f24c36c2700 x1636708563213008/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026148 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 20 03:22:28 fir-md1-s1 kernel: Lustre: 20460:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jun 20 03:22:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 804bb2d0-a656-6c01-b0db-5b53058fb0f9 (at 10.8.9.9@o2ib6) reconnecting Jun 20 03:22:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 20 03:22:42 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026155/real 1561026155] req@ffff8f1c772f4800 x1636708563212800/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026162 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 20 03:22:42 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 20 03:22:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 804bb2d0-a656-6c01-b0db-5b53058fb0f9 (at 10.8.9.9@o2ib6) reconnecting Jun 20 03:22:56 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 03:22:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 20 03:22:56 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 03:23:03 fir-md1-s1 kernel: Lustre: 22289:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026176/real 1561026176] req@ffff8f1e30743c00 x1636708563213088/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026183 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 20 03:23:03 fir-md1-s1 kernel: Lustre: 22289:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jun 20 03:23:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 804bb2d0-a656-6c01-b0db-5b53058fb0f9 (at 10.8.9.9@o2ib6) reconnecting Jun 20 03:23:17 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 03:23:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 20 03:23:17 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 03:23:38 fir-md1-s1 kernel: Lustre: 20460:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026211/real 1561026211] req@ffff8f24c36c2700 x1636708563213008/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026218 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 20 03:23:38 fir-md1-s1 kernel: Lustre: 20460:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jun 20 03:23:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 804bb2d0-a656-6c01-b0db-5b53058fb0f9 (at 10.8.9.9@o2ib6) reconnecting Jun 20 03:23:38 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 03:23:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 20 03:23:38 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 03:23:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 804bb2d0-a656-6c01-b0db-5b53058fb0f9 (at 10.8.9.9@o2ib6) reconnecting Jun 20 03:23:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 03:23:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 20 03:23:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 03:24:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 804bb2d0-a656-6c01-b0db-5b53058fb0f9 (at 10.8.9.9@o2ib6) reconnecting Jun 20 03:24:41 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 20 03:24:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 20 03:24:41 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 20 03:24:48 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561026281/real 1561026281] req@ffff8f1c772f4800 x1636708563212800/t0(0) o106->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561026288 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 20 03:24:48 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 27 previous similar messages Jun 20 03:24:48 fir-md1-s1 kernel: LustreError: 20460:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.8@o2ib6) returned error from glimpse AST (req@ffff8f24c36c2700 x1636708563213008 status -107 rc -107), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f2376712f40/0x5d9ee61d1db84dae lrc: 4/0,0 mode: PW/PW res: [0x200025f94:0x1628f:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x40200000000000 nid: 10.8.9.8@o2ib6 remote: 0xb7f6b0a5194d419f expref: 59 pid: 21433 timeout: 0 lvb_type: 0 Jun 20 03:24:48 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.9.8@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Jun 20 03:24:48 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 197s: evicting client at 10.8.9.8@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1330a91680/0x5d9ee61d1db8506a lrc: 4/0,0 mode: PW/PW res: [0x200025b09:0x2431:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x40200000000000 nid: 10.8.9.8@o2ib6 remote: 0xb7f6b0a5194d444d expref: 60 pid: 26257 timeout: 0 lvb_type: 0 Jun 20 03:24:48 fir-md1-s1 kernel: LustreError: 20460:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Jun 20 03:25:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 2a12b0b1-96b1-b609-eece-2f0222928c53 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2507b0a000, cur 1561026330 expire 1561026180 last 1561026103 Jun 20 03:25:30 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 20 03:28:39 fir-md1-s1 kernel: LustreError: 21710:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 86016 GRANT, real grant 0 Jun 20 03:28:39 fir-md1-s1 kernel: LustreError: 21710:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 275 previous similar messages Jun 20 03:38:49 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 03:38:49 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 20 03:48:54 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 03:48:54 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 760 previous similar messages Jun 20 03:58:56 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 03:58:56 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 716 previous similar messages Jun 20 04:08:56 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 57344 GRANT, real grant 0 Jun 20 04:08:56 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2561 previous similar messages Jun 20 04:18:57 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 04:18:57 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 706 previous similar messages Jun 20 04:28:58 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 04:28:58 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 577 previous similar messages Jun 20 04:39:06 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 04:39:06 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 919 previous similar messages Jun 20 04:49:14 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 04:49:14 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1034 previous similar messages Jun 20 04:59:17 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 04:59:17 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1061 previous similar messages Jun 20 05:09:20 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 20 05:09:20 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 923 previous similar messages Jun 20 05:19:34 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 20 05:19:34 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 825 previous similar messages Jun 20 05:29:36 fir-md1-s1 kernel: LustreError: 21710:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 05:29:36 fir-md1-s1 kernel: LustreError: 21710:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1005 previous similar messages Jun 20 05:39:40 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 20 05:39:40 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 743 previous similar messages Jun 20 05:49:44 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 05:49:44 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 959 previous similar messages Jun 20 05:59:50 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 05:59:50 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 781 previous similar messages Jun 20 06:09:57 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 06:09:57 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1045 previous similar messages Jun 20 06:19:58 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 118784 GRANT, real grant 0 Jun 20 06:19:58 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 892 previous similar messages Jun 20 06:30:06 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 06:30:06 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 640 previous similar messages Jun 20 06:40:11 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 57344 GRANT, real grant 0 Jun 20 06:40:11 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 524 previous similar messages Jun 20 06:50:16 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 06:50:16 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 494 previous similar messages Jun 20 07:00:22 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 07:00:22 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 542 previous similar messages Jun 20 07:10:25 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 07:10:25 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 727 previous similar messages Jun 20 07:20:27 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 07:20:27 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 776 previous similar messages Jun 20 07:30:29 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 07:30:29 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 700 previous similar messages Jun 20 07:40:33 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 07:40:33 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 687 previous similar messages Jun 20 07:50:39 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 135168 GRANT, real grant 0 Jun 20 07:50:39 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 732 previous similar messages Jun 20 08:00:43 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 08:00:43 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 536 previous similar messages Jun 20 08:10:43 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 08:10:43 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 616 previous similar messages Jun 20 08:20:46 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 49152 GRANT, real grant 0 Jun 20 08:20:46 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 703 previous similar messages Jun 20 08:30:56 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 08:30:56 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 681 previous similar messages Jun 20 08:40:59 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 08:40:59 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 58668 previous similar messages Jun 20 08:51:01 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 20 08:51:01 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1290 previous similar messages Jun 20 09:02:49 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 09:02:49 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 521 previous similar messages Jun 20 09:18:28 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 09:18:28 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jun 20 09:31:48 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 09:31:48 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 32 previous similar messages Jun 20 09:46:56 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 09:46:56 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 10 previous similar messages Jun 20 09:56:57 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 09:56:57 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1835 previous similar messages Jun 20 10:07:00 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 10:07:00 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 20 10:17:08 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 10:17:08 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 735 previous similar messages Jun 20 10:27:10 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 10:27:10 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 787 previous similar messages Jun 20 10:37:18 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 86016 GRANT, real grant 0 Jun 20 10:37:18 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 51449 previous similar messages Jun 20 10:47:19 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 73728 GRANT, real grant 0 Jun 20 10:47:19 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7765 previous similar messages Jun 20 10:57:41 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 102400 GRANT, real grant 0 Jun 20 10:57:41 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 246 previous similar messages Jun 20 11:08:11 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 11:08:11 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 226 previous similar messages Jun 20 11:18:18 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 11:18:18 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 236 previous similar messages Jun 20 11:28:22 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 86016 GRANT, real grant 0 Jun 20 11:28:22 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2275 previous similar messages Jun 20 11:38:24 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 98304 GRANT, real grant 0 Jun 20 11:38:24 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 590 previous similar messages Jun 20 11:48:26 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 11:48:26 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 684 previous similar messages Jun 20 11:58:31 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 11:58:31 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 912 previous similar messages Jun 20 12:08:33 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 12:08:33 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 58823 previous similar messages Jun 20 12:08:50 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 12:08:50 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages Jun 20 12:18:33 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 12:18:33 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1245 previous similar messages Jun 20 12:28:33 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 12:28:33 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 230 previous similar messages Jun 20 12:38:34 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 12:38:34 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 697 previous similar messages Jun 20 12:48:34 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 12:48:34 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 771 previous similar messages Jun 20 12:58:35 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 20 12:58:35 fir-md1-s1 kernel: LustreError: 21293:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 879 previous similar messages Jun 20 13:08:36 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 20 13:08:36 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1448 previous similar messages Jun 20 13:18:38 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 20 13:18:38 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1775 previous similar messages Jun 20 13:28:38 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 13:28:38 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1863 previous similar messages Jun 20 13:58:35 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 13:58:35 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 664 previous similar messages Jun 20 14:00:47 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 14:00:47 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 20 14:27:25 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 14:29:37 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 14:29:37 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 20 14:56:14 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 14:56:19 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 20 14:58:27 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 15:24:30 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 15:24:43 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 15:25:04 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 15:49:49 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 15:49:49 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jun 20 15:50:19 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 20 15:52:15 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 15:54:07 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 15:54:07 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 20 15:55:45 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:02:11 fir-md1-s1 kernel: LustreError: 25998:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 16:04:14 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:05:33 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:06:31 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:14:02 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 16:19:26 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:19:39 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:20:07 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:20:07 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages Jun 20 16:23:50 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:25:51 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 16:26:54 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:27:43 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:29:25 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:32:15 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:32:15 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 20 16:37:41 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 16:39:27 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:39:27 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages Jun 20 16:49:31 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 16:49:31 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages Jun 20 16:49:36 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 16:55:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 815d7676-5c34-1cc9-c5dd-bad0fb6e70bb (at 10.8.14.8@o2ib6) Jun 20 16:55:30 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jun 20 16:55:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d8cec7bd-0c71-5918-8514-07b7e416bc71 (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4518b05c00, cur 1561074934 expire 1561074784 last 1561074707 Jun 20 16:55:34 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 20 16:55:36 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 815d7676-5c34-1cc9-c5dd-bad0fb6e70bb (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25257d9800, cur 1561074936 expire 1561074786 last 1561074709 Jun 20 17:01:27 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 17:02:07 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 17:02:07 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages Jun 20 17:12:20 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 17:12:20 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages Jun 20 17:13:14 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 17:22:21 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 17:22:21 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages Jun 20 17:25:06 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 17:32:52 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 17:32:52 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages Jun 20 17:36:31 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 17:43:31 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 17:43:31 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 184 previous similar messages Jun 20 17:48:22 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 17:53:46 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 17:53:46 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages Jun 20 18:00:13 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 18:04:02 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 18:04:02 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 85 previous similar messages Jun 20 18:12:12 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 18:15:32 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 18:15:32 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 48 previous similar messages Jun 20 18:23:57 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 18:26:00 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 18:26:00 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages Jun 20 18:36:11 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 18:37:35 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 18:37:35 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages Jun 20 18:47:33 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 18:49:22 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 18:49:22 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 37 previous similar messages Jun 20 18:59:21 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 18:59:39 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 18:59:39 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages Jun 20 19:10:58 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 19:10:58 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages Jun 20 19:11:36 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 19:21:30 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 19:21:30 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 33 previous similar messages Jun 20 19:23:25 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 19:35:14 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 19:38:18 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 19:38:18 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages Jun 20 19:47:04 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 19:57:52 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 19:57:52 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages Jun 20 19:58:52 fir-md1-s1 kernel: LustreError: 21717:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 20:10:41 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 20:16:38 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 20:16:38 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 20 20:22:28 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 20:34:16 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 20:46:08 fir-md1-s1 kernel: LustreError: 27482:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 20:58:00 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 21:09:51 fir-md1-s1 kernel: LustreError: 21717:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 21:21:35 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 21:33:18 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 21:44:26 fir-md1-s1 kernel: LustreError: 21717:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 21:45:46 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 21:45:46 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages Jun 20 21:55:32 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 22:04:30 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:06:09 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 22:16:51 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 22:19:43 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:20:34 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:21:50 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:26:13 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:27:15 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 22:27:33 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:31:33 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:31:33 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 20 22:34:01 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:36:01 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:36:01 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 20 22:37:40 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 22:42:17 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:47:57 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 22:48:13 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:48:13 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 83 previous similar messages Jun 20 22:58:13 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 22:58:33 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 22:58:33 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages Jun 20 23:08:23 fir-md1-s1 kernel: LustreError: 25998:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 23:09:44 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 23:09:44 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 184 previous similar messages Jun 20 23:18:34 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 20 23:22:00 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 23:22:00 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 28 previous similar messages Jun 20 23:32:15 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 23:32:15 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 31 previous similar messages Jun 20 23:42:48 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 23:42:48 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages Jun 20 23:53:11 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 20 23:53:11 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 107 previous similar messages Jun 21 00:03:48 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 00:03:48 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 29 previous similar messages Jun 21 00:13:52 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 00:13:52 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 67 previous similar messages Jun 21 00:24:09 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 00:24:09 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 279 previous similar messages Jun 21 00:34:17 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 00:34:17 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 76 previous similar messages Jun 21 00:44:28 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 00:44:28 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 237 previous similar messages Jun 21 00:54:33 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 00:54:33 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 604 previous similar messages Jun 21 00:57:09 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 00:57:15 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 00:57:21 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 00:57:23 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 00:57:23 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jun 21 00:57:27 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 00:57:27 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 31 previous similar messages Jun 21 00:57:41 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 21 00:57:41 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jun 21 00:57:57 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 57344 GRANT, real grant 0 Jun 21 00:57:57 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 803 previous similar messages Jun 21 00:58:37 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 21 00:58:37 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jun 21 01:01:46 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 61440 GRANT, real grant 0 Jun 21 01:01:46 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 163 previous similar messages Jun 21 01:04:36 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 01:04:36 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 111 previous similar messages Jun 21 01:14:48 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 01:14:48 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 666 previous similar messages Jun 21 01:16:27 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 01:16:56 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 01:16:56 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jun 21 01:17:38 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 106496 GRANT, real grant 0 Jun 21 01:17:38 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 35 previous similar messages Jun 21 01:18:48 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 21 01:18:48 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 92 previous similar messages Jun 21 01:20:59 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 01:20:59 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 70 previous similar messages Jun 21 01:24:49 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 01:24:49 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1375 previous similar messages Jun 21 01:25:25 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 01:25:25 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 74 previous similar messages Jun 21 01:34:02 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 01:34:02 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 438 previous similar messages Jun 21 01:34:49 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 01:34:49 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 384 previous similar messages Jun 21 01:44:06 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 01:44:06 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 503 previous similar messages Jun 21 01:44:52 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 01:44:52 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 884 previous similar messages Jun 21 01:54:16 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 01:54:16 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 21 01:54:53 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 01:54:53 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1624 previous similar messages Jun 21 02:04:21 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 02:04:21 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 478 previous similar messages Jun 21 02:04:54 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 02:04:54 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 296 previous similar messages Jun 21 02:14:31 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 02:14:31 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 470 previous similar messages Jun 21 02:15:03 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 02:15:03 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3707 previous similar messages Jun 21 02:24:35 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 02:24:35 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 21 02:25:08 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 02:25:08 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2273 previous similar messages Jun 21 02:34:44 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 02:34:44 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1632 previous similar messages Jun 21 02:35:22 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 02:35:22 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 330 previous similar messages Jun 21 02:44:51 fir-md1-s1 kernel: LustreError: 25998:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 02:44:51 fir-md1-s1 kernel: LustreError: 25998:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 421 previous similar messages Jun 21 02:45:29 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 02:45:29 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1489 previous similar messages Jun 21 02:55:31 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 02:55:31 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1643 previous similar messages Jun 21 02:55:55 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 02:55:55 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 49 previous similar messages Jun 21 03:05:32 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 03:05:32 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 259 previous similar messages Jun 21 03:05:57 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 03:05:57 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 32 previous similar messages Jun 21 03:15:37 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 03:15:37 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1410 previous similar messages Jun 21 03:16:05 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 126976 GRANT, real grant 0 Jun 21 03:16:05 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1263 previous similar messages Jun 21 03:25:37 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 03:25:37 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2751 previous similar messages Jun 21 03:26:08 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 03:26:08 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 243 previous similar messages Jun 21 03:35:56 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 03:35:56 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 288 previous similar messages Jun 21 03:36:09 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 03:36:09 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 383 previous similar messages Jun 21 03:45:59 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 03:45:59 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1828 previous similar messages Jun 21 03:46:10 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 03:46:10 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 716 previous similar messages Jun 21 03:55:59 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 03:55:59 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2206 previous similar messages Jun 21 03:56:10 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 03:56:10 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 591 previous similar messages Jun 21 04:06:00 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 04:06:00 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 255 previous similar messages Jun 21 04:06:13 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 04:06:13 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1670 previous similar messages Jun 21 04:16:01 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 04:16:01 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3284 previous similar messages Jun 21 04:16:14 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 04:16:14 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1822 previous similar messages Jun 21 04:26:02 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 04:26:02 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1023 previous similar messages Jun 21 04:26:15 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 04:26:15 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 530 previous similar messages Jun 21 04:36:03 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 04:36:03 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 703 previous similar messages Jun 21 04:36:17 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 04:36:17 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 883 previous similar messages Jun 21 04:46:03 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 04:46:03 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5430 previous similar messages Jun 21 04:46:25 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 04:46:25 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1018 previous similar messages Jun 21 04:56:05 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 04:56:05 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 384 previous similar messages Jun 21 04:56:29 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 04:56:29 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 976 previous similar messages Jun 21 05:06:06 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 05:06:06 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4767 previous similar messages Jun 21 05:06:32 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 106496 GRANT, real grant 0 Jun 21 05:06:32 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 661 previous similar messages Jun 21 05:16:09 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 05:16:09 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3850 previous similar messages Jun 21 05:16:49 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 05:16:49 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 898 previous similar messages Jun 21 05:26:09 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 05:26:09 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 376 previous similar messages Jun 21 05:26:51 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 05:26:51 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 969 previous similar messages Jun 21 05:36:11 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 05:36:11 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4162 previous similar messages Jun 21 05:36:53 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 05:36:53 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1049 previous similar messages Jun 21 05:46:15 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 05:46:15 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2518 previous similar messages Jun 21 05:47:01 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 05:47:01 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1028 previous similar messages Jun 21 05:56:16 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 05:56:16 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 459 previous similar messages Jun 21 05:57:15 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 05:57:15 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 732 previous similar messages Jun 21 06:06:16 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 06:06:16 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3435 previous similar messages Jun 21 06:07:26 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 06:07:26 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1041 previous similar messages Jun 21 06:16:16 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 06:16:16 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3560 previous similar messages Jun 21 06:17:30 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 06:17:30 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 834 previous similar messages Jun 21 06:26:25 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 06:26:25 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 410 previous similar messages Jun 21 06:27:34 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 06:27:34 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 551 previous similar messages Jun 21 06:36:28 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 06:36:28 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3289 previous similar messages Jun 21 06:37:34 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 06:37:34 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 309 previous similar messages Jun 21 06:46:30 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 06:46:30 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3608 previous similar messages Jun 21 06:47:35 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 06:47:35 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 524 previous similar messages Jun 21 06:56:31 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 06:56:31 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 370 previous similar messages Jun 21 06:57:35 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 06:57:35 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 755 previous similar messages Jun 21 07:06:33 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 07:06:33 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2292 previous similar messages Jun 21 07:07:39 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 07:07:39 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 746 previous similar messages Jun 21 07:16:34 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 07:16:34 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5203 previous similar messages Jun 21 07:17:39 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 07:17:39 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 675 previous similar messages Jun 21 07:26:36 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 07:26:36 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 466 previous similar messages Jun 21 07:27:49 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 07:27:49 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 689 previous similar messages Jun 21 07:36:48 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 07:36:48 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2843 previous similar messages Jun 21 07:37:53 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 07:37:53 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 717 previous similar messages Jun 21 07:46:49 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 07:46:49 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2442 previous similar messages Jun 21 07:47:59 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 135168 GRANT, real grant 0 Jun 21 07:47:59 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 696 previous similar messages Jun 21 07:56:55 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 07:56:55 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 519 previous similar messages Jun 21 07:57:59 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 81920 GRANT, real grant 0 Jun 21 07:57:59 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 438 previous similar messages Jun 21 08:06:55 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 08:06:55 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2184 previous similar messages Jun 21 08:08:07 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 08:08:07 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 539 previous similar messages Jun 21 08:16:56 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 08:16:56 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3160 previous similar messages Jun 21 08:18:11 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 57344 GRANT, real grant 0 Jun 21 08:18:11 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 925 previous similar messages Jun 21 08:27:00 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 08:27:00 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 350 previous similar messages Jun 21 08:28:25 fir-md1-s1 kernel: LustreError: 25998:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 08:28:25 fir-md1-s1 kernel: LustreError: 25998:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 756 previous similar messages Jun 21 08:37:06 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 08:37:06 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4244 previous similar messages Jun 21 08:38:25 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 08:38:25 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 58533 previous similar messages Jun 21 08:47:15 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 08:47:15 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3126 previous similar messages Jun 21 08:48:28 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 21 08:48:28 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1221 previous similar messages Jun 21 08:57:15 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 08:57:15 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 325 previous similar messages Jun 21 09:02:18 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 09:02:18 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 588 previous similar messages Jun 21 09:07:16 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 09:07:16 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2868 previous similar messages Jun 21 09:17:17 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 09:17:17 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2968 previous similar messages Jun 21 09:17:25 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 09:17:25 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jun 21 09:27:19 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 09:27:19 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 618 previous similar messages Jun 21 09:30:54 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 09:30:54 fir-md1-s1 kernel: LustreError: 20504:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 32 previous similar messages Jun 21 09:37:20 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 09:37:20 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2752 previous similar messages Jun 21 09:47:22 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 09:47:22 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2202 previous similar messages Jun 21 09:57:23 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 09:57:23 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 577 previous similar messages Jun 21 10:05:13 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 21 10:05:13 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jun 21 10:07:27 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 10:07:27 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3400 previous similar messages Jun 21 10:17:31 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 10:17:31 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1640 previous similar messages Jun 21 10:27:38 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 10:27:38 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2409 previous similar messages Jun 21 10:37:40 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 10:37:40 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2335 previous similar messages Jun 21 10:47:42 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 10:47:42 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 557 previous similar messages Jun 21 10:57:45 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 10:57:45 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2457 previous similar messages Jun 21 11:06:44 fir-md1-s1 kernel: Lustre: 21073:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0e18e9a100 x1635087235946768/t0(0) o36->c50a2569-5f68-c0c4-a8b8-bfb61fe4dbbb@10.9.114.5@o2ib4:19/0 lens 536/2888 e 1 to 0 dl 1561140409 ref 2 fl Interpret:/0/0 rc 0/0 Jun 21 11:06:44 fir-md1-s1 kernel: Lustre: 21073:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jun 21 11:06:46 fir-md1-s1 kernel: Lustre: 21073:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0e234bc800 x1634124367904096/t0(0) o36->190e8c90-938d-b7f6-84df-7662b8e78e53@10.9.107.71@o2ib4:21/0 lens 552/2888 e 1 to 0 dl 1561140411 ref 2 fl Interpret:/0/0 rc 0/0 Jun 21 11:06:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client c50a2569-5f68-c0c4-a8b8-bfb61fe4dbbb (at 10.9.114.5@o2ib4) reconnecting Jun 21 11:06:50 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 21 11:06:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a99d6390-552e-efef-43b1-60bd87733129 (at 10.9.114.5@o2ib4) Jun 21 11:06:50 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 21 11:06:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 8ba50a96-f3d9-3920-760c-8aedb752cbea (at 10.9.107.71@o2ib4) Jun 21 11:07:41 fir-md1-s1 kernel: LNetError: 20180:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Jun 21 11:07:41 fir-md1-s1 kernel: LNetError: 20180:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (105): c: 7, oc: 0, rc: 8 Jun 21 11:09:27 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14c7777000, cur 1561140567 expire 1561140417 last 1561140340 Jun 21 11:09:27 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 21 11:09:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 1675e4f5-80cb-6029-9271-7b3f4a7873d6 (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2532670000, cur 1561140583 expire 1561140433 last 1561140356 Jun 21 11:10:25 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 11:10:25 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jun 21 11:10:51 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 11:10:51 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jun 21 11:11:22 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 11:11:22 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jun 21 11:12:06 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 110592 GRANT, real grant 0 Jun 21 11:12:06 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1585 previous similar messages Jun 21 11:13:27 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 21 11:13:27 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 209 previous similar messages Jun 21 11:13:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 21 11:15:21 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 11:15:21 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 157 previous similar messages Jun 21 11:16:07 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 61440 GRANT, real grant 0 Jun 21 11:16:07 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 21 11:21:08 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 11:21:08 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 21 11:31:17 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 11:31:17 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 406 previous similar messages Jun 21 11:40:59 fir-md1-s1 kernel: LNetError: 20180:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 2 seconds Jun 21 11:40:59 fir-md1-s1 kernel: LNetError: 20180:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (108): c: 8, oc: 0, rc: 8 Jun 21 11:41:19 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 11:41:19 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 550 previous similar messages Jun 21 11:47:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 21 11:47:51 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 21 11:51:07 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 11:51:07 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5477 previous similar messages Jun 21 11:51:35 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 21 11:51:35 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 20361 previous similar messages Jun 21 11:52:22 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 11:52:22 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3381 previous similar messages Jun 21 11:54:53 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 11:54:53 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1585 previous similar messages Jun 21 12:00:15 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 12:00:15 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 95 previous similar messages Jun 21 12:10:16 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 12:10:16 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1504 previous similar messages Jun 21 12:20:21 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 12:20:21 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5067 previous similar messages Jun 21 12:30:24 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 12:30:24 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 689 previous similar messages Jun 21 12:40:24 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 12:40:24 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5807 previous similar messages Jun 21 12:50:25 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 12:50:25 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2656 previous similar messages Jun 21 13:00:26 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 13:00:26 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6064 previous similar messages Jun 21 13:10:28 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 13:10:28 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4237 previous similar messages Jun 21 13:20:28 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 13:20:28 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5924 previous similar messages Jun 21 13:30:40 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 13:30:40 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4528 previous similar messages Jun 21 13:40:42 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 13:40:42 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5875 previous similar messages Jun 21 13:50:44 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 13:50:44 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 431 previous similar messages Jun 21 14:00:53 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 14:00:53 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3226 previous similar messages Jun 21 14:10:54 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 14:10:54 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2729 previous similar messages Jun 21 14:20:56 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 14:20:56 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2474 previous similar messages Jun 21 14:31:14 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 14:31:14 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9037 previous similar messages Jun 21 14:41:21 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 14:41:21 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 309 previous similar messages Jun 21 14:51:29 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 14:51:29 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 461 previous similar messages Jun 21 15:01:32 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 15:01:32 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6161 previous similar messages Jun 21 15:11:35 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 15:11:35 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 269 previous similar messages Jun 21 15:21:37 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 15:21:37 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4361 previous similar messages Jun 21 15:31:42 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 15:31:42 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5741 previous similar messages Jun 21 15:41:52 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 15:41:52 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 380 previous similar messages Jun 21 15:52:24 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 15:52:24 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4143 previous similar messages Jun 21 16:02:25 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 16:02:25 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4240 previous similar messages Jun 21 16:12:27 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 16:12:27 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1721 previous similar messages Jun 21 16:22:36 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 16:22:36 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4798 previous similar messages Jun 21 16:32:37 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 16:32:37 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 370 previous similar messages Jun 21 16:43:00 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 16:43:00 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5984 previous similar messages Jun 21 16:53:02 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 16:53:02 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 374 previous similar messages Jun 21 17:03:11 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 17:03:11 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6833 previous similar messages Jun 21 17:13:12 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 17:13:12 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6018 previous similar messages Jun 21 17:23:13 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 17:23:13 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 460 previous similar messages Jun 21 17:33:14 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 17:33:14 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3622 previous similar messages Jun 21 17:43:14 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 17:43:14 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1956 previous similar messages Jun 21 17:53:29 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 17:53:29 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 827 previous similar messages Jun 21 18:03:39 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 18:03:39 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1131 previous similar messages Jun 21 18:13:40 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 18:13:40 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1681 previous similar messages Jun 21 18:20:28 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 18:20:28 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 38827 previous similar messages Jun 21 18:21:46 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 57344 GRANT, real grant 0 Jun 21 18:21:46 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1603 previous similar messages Jun 21 18:24:24 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 18:24:24 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2154 previous similar messages Jun 21 18:26:00 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 61440 GRANT, real grant 0 Jun 21 18:26:00 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 214 previous similar messages Jun 21 18:31:03 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 21 18:31:03 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 21 18:34:39 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 18:34:39 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3335 previous similar messages Jun 21 18:41:04 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 18:41:04 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 413 previous similar messages Jun 21 18:44:55 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 18:44:55 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 255 previous similar messages Jun 21 18:51:06 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 21 18:51:06 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 564 previous similar messages Jun 21 18:54:57 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 18:54:57 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2306 previous similar messages Jun 21 19:01:22 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 94208 GRANT, real grant 0 Jun 21 19:01:22 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15785 previous similar messages Jun 21 19:05:20 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 19:05:20 fir-md1-s1 kernel: Lustre: 21668:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2455 previous similar messages Jun 21 19:15:25 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 19:15:25 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 281 previous similar messages Jun 21 19:25:42 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 19:25:42 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2577 previous similar messages Jun 21 19:36:06 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 19:36:06 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3780 previous similar messages Jun 21 19:46:10 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 19:46:10 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 267 previous similar messages Jun 21 19:56:37 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 19:56:37 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1344 previous similar messages Jun 21 20:06:44 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 20:06:44 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1396 previous similar messages Jun 21 20:16:54 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 20:16:54 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 299 previous similar messages Jun 21 20:26:57 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 20:26:57 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 278 previous similar messages Jun 21 20:36:59 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 20:36:59 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1285 previous similar messages Jun 21 20:47:01 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 20:47:01 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2576 previous similar messages Jun 21 20:57:07 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 20:57:07 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 624 previous similar messages Jun 21 21:07:10 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 21:07:10 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3192 previous similar messages Jun 21 21:17:10 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 21:17:10 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2201 previous similar messages Jun 21 21:27:12 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 21:27:12 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 887 previous similar messages Jun 21 21:37:14 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 21:37:14 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2216 previous similar messages Jun 21 21:47:25 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 21:47:25 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1435 previous similar messages Jun 21 21:57:40 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 21:57:40 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 273 previous similar messages Jun 21 22:08:02 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 22:08:02 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 625 previous similar messages Jun 21 22:18:08 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 22:18:08 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1774 previous similar messages Jun 21 22:28:34 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 22:28:34 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 359 previous similar messages Jun 21 22:38:38 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 22:38:38 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3257 previous similar messages Jun 21 22:48:38 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 22:48:38 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2435 previous similar messages Jun 21 22:58:49 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 22:58:49 fir-md1-s1 kernel: Lustre: 21669:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 319 previous similar messages Jun 21 23:08:57 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 23:08:57 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 485 previous similar messages Jun 21 23:11:29 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 2fbdd3a1-1348-387a-9c62-8e4888f673df (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34ea3d8800, cur 1561183889 expire 1561183739 last 1561183662 Jun 21 23:11:29 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 21 23:11:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d80b2c48-58e4-12d3-5b26-0e7343b58644 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f1aed7c00, cur 1561183894 expire 1561183744 last 1561183667 Jun 21 23:11:34 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 21 23:11:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ec76f1db-9c9b-bbe0-847f-90a9d517c8dc (at 10.8.9.8@o2ib6) Jun 21 23:11:44 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 21 23:18:59 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 23:18:59 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2494 previous similar messages Jun 21 23:28:59 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 23:28:59 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4301 previous similar messages Jun 21 23:39:01 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 23:39:01 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5072 previous similar messages Jun 21 23:49:01 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 23:49:01 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5445 previous similar messages Jun 21 23:59:02 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 21 23:59:02 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4972 previous similar messages Jun 22 00:09:02 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 00:09:02 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3593 previous similar messages Jun 22 00:19:02 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 00:19:02 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3499 previous similar messages Jun 22 00:29:03 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 00:29:03 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3488 previous similar messages Jun 22 00:39:03 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 00:39:03 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3496 previous similar messages Jun 22 00:49:04 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 00:49:04 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4026 previous similar messages Jun 22 00:57:11 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 00:57:11 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 43417 previous similar messages Jun 22 00:58:47 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 00:58:47 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1937 previous similar messages Jun 22 00:59:05 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 00:59:05 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4224 previous similar messages Jun 22 01:01:46 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 61440 GRANT, real grant 0 Jun 22 01:01:46 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 125 previous similar messages Jun 22 01:09:06 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 01:09:06 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 606 previous similar messages Jun 22 01:16:58 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 01:16:58 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 22 01:17:39 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 22 01:17:39 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jun 22 01:18:54 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 57344 GRANT, real grant 0 Jun 22 01:18:54 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 87 previous similar messages Jun 22 01:19:07 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 01:19:07 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2365 previous similar messages Jun 22 01:21:26 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 01:21:26 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 273 previous similar messages Jun 22 01:26:30 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 49152 GRANT, real grant 0 Jun 22 01:26:30 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 280 previous similar messages Jun 22 01:29:07 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 01:29:07 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4021 previous similar messages Jun 22 01:36:33 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 01:36:33 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 632 previous similar messages Jun 22 01:39:07 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 01:39:07 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4213 previous similar messages Jun 22 01:46:36 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 01:46:36 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 959 previous similar messages Jun 22 01:49:07 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 01:49:07 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4191 previous similar messages Jun 22 01:56:39 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 01:56:39 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 656 previous similar messages Jun 22 01:59:07 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 01:59:07 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4250 previous similar messages Jun 22 02:06:46 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 02:06:46 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1005 previous similar messages Jun 22 02:09:09 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 02:09:09 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3816 previous similar messages Jun 22 02:16:55 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 02:16:55 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 657 previous similar messages Jun 22 02:19:09 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 02:19:09 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4170 previous similar messages Jun 22 02:27:02 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 02:27:02 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2118 previous similar messages Jun 22 02:29:10 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 02:29:10 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3770 previous similar messages Jun 22 02:37:06 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 02:37:06 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 646 previous similar messages Jun 22 02:39:13 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 02:39:13 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3238 previous similar messages Jun 22 02:47:09 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 02:47:09 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 947 previous similar messages Jun 22 02:49:13 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 02:49:13 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3264 previous similar messages Jun 22 02:57:11 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 02:57:11 fir-md1-s1 kernel: LustreError: 21716:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 426 previous similar messages Jun 22 02:59:13 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 02:59:13 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3218 previous similar messages Jun 22 03:07:12 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 03:07:12 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 448 previous similar messages Jun 22 03:09:14 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 03:09:14 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3552 previous similar messages Jun 22 03:17:13 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 03:17:13 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1420 previous similar messages Jun 22 03:19:15 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 03:19:15 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3482 previous similar messages Jun 22 03:27:14 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 03:27:14 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 629 previous similar messages Jun 22 03:29:15 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 03:29:15 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3227 previous similar messages Jun 22 03:37:15 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 03:37:15 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 726 previous similar messages Jun 22 03:39:15 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 03:39:15 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2921 previous similar messages Jun 22 03:47:17 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 03:47:17 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 872 previous similar messages Jun 22 03:49:16 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 03:49:16 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2604 previous similar messages Jun 22 03:57:19 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 03:57:19 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1046 previous similar messages Jun 22 03:59:16 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 03:59:16 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2810 previous similar messages Jun 22 04:07:22 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 04:07:22 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 791 previous similar messages Jun 22 04:09:17 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 04:09:17 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2460 previous similar messages Jun 22 04:17:23 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 28672 GRANT, real grant 0 Jun 22 04:17:23 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1540 previous similar messages Jun 22 04:19:17 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 04:19:17 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2654 previous similar messages Jun 22 04:25:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f15da91-4546-507e-8c99-9e08b5e219a4 (at 10.8.15.10@o2ib6) Jun 22 04:25:36 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 22 04:26:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 768e69f1-686d-dc63-c888-d7b2745331f7 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25211b8400, cur 1561202785 expire 1561202635 last 1561202558 Jun 22 04:26:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 768e69f1-686d-dc63-c888-d7b2745331f7 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fdaacc00, cur 1561202803 expire 1561202653 last 1561202576 Jun 22 04:26:43 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 22 04:27:32 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 04:27:32 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1895 previous similar messages Jun 22 04:29:18 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 04:29:18 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3076 previous similar messages Jun 22 04:37:33 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 81920 GRANT, real grant 0 Jun 22 04:37:33 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 435 previous similar messages Jun 22 04:39:18 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 04:39:18 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3391 previous similar messages Jun 22 04:47:37 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 04:47:37 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 626 previous similar messages Jun 22 04:49:19 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 04:49:19 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3215 previous similar messages Jun 22 04:57:44 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 04:57:44 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1073 previous similar messages Jun 22 04:59:19 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 04:59:19 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3782 previous similar messages Jun 22 05:07:55 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 05:07:55 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1176 previous similar messages Jun 22 05:09:20 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 05:09:20 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3081 previous similar messages Jun 22 05:18:00 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 05:18:00 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 839 previous similar messages Jun 22 05:19:21 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 05:19:21 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2657 previous similar messages Jun 22 05:28:00 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 05:28:00 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1085 previous similar messages Jun 22 05:29:22 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 05:29:22 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3095 previous similar messages Jun 22 05:38:07 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 05:38:07 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1089 previous similar messages Jun 22 05:38:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 2c97d373-364e-c157-5583-02820de3bb2e (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148c8cac00, cur 1561207121 expire 1561206971 last 1561206894 Jun 22 05:39:22 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 05:39:22 fir-md1-s1 kernel: Lustre: 20458:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1902 previous similar messages Jun 22 05:48:11 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 22 05:48:11 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 999 previous similar messages Jun 22 05:49:23 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 05:49:23 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2890 previous similar messages Jun 22 05:58:13 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 05:58:13 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 946 previous similar messages Jun 22 05:59:23 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 05:59:23 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4018 previous similar messages Jun 22 06:08:15 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 06:08:15 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 840 previous similar messages Jun 22 06:09:24 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 06:09:24 fir-md1-s1 kernel: Lustre: 21417:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5540 previous similar messages Jun 22 06:18:24 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 06:18:24 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1077 previous similar messages Jun 22 06:19:27 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 06:19:27 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6484 previous similar messages Jun 22 06:28:27 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 06:28:27 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 976 previous similar messages Jun 22 06:29:27 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 06:29:27 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5334 previous similar messages Jun 22 06:38:29 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 06:38:29 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1047 previous similar messages Jun 22 06:39:28 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 06:39:28 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6116 previous similar messages Jun 22 06:48:30 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 53248 GRANT, real grant 0 Jun 22 06:48:30 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1075 previous similar messages Jun 22 06:49:28 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 06:49:28 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4963 previous similar messages Jun 22 06:58:31 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 49152 GRANT, real grant 0 Jun 22 06:58:31 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 563 previous similar messages Jun 22 06:59:31 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 06:59:31 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1915 previous similar messages Jun 22 07:08:36 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 07:08:36 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 568 previous similar messages Jun 22 07:09:31 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 07:09:31 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3761 previous similar messages Jun 22 07:18:36 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 07:18:36 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 730 previous similar messages Jun 22 07:19:32 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 07:19:32 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3753 previous similar messages Jun 22 07:28:38 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 07:28:38 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 746 previous similar messages Jun 22 07:29:32 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 07:29:32 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3204 previous similar messages Jun 22 07:38:38 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 07:38:38 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 651 previous similar messages Jun 22 07:48:38 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 07:48:38 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 692 previous similar messages Jun 22 07:58:49 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 22 07:58:49 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 723 previous similar messages Jun 22 08:08:50 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 08:08:50 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 477 previous similar messages Jun 22 08:18:56 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 22 08:18:56 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 482 previous similar messages Jun 22 08:25:14 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 08:25:14 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3181 previous similar messages Jun 22 08:29:09 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 49152 GRANT, real grant 0 Jun 22 08:29:09 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 969 previous similar messages Jun 22 08:33:06 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 08:36:33 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 08:36:33 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 22 08:39:16 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 08:39:16 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 710 previous similar messages Jun 22 08:42:28 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 08:42:28 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages Jun 22 08:49:17 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 08:49:17 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 58538 previous similar messages Jun 22 08:52:31 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 08:52:31 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 647 previous similar messages Jun 22 08:59:21 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 69632 GRANT, real grant 0 Jun 22 08:59:21 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1355 previous similar messages Jun 22 09:02:32 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 09:02:32 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2533 previous similar messages Jun 22 09:11:31 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 09:11:31 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 412 previous similar messages Jun 22 09:12:34 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 09:12:34 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1433 previous similar messages Jun 22 09:22:34 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 09:22:34 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2123 previous similar messages Jun 22 09:27:26 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 09:27:26 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jun 22 09:32:36 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 09:32:36 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2710 previous similar messages Jun 22 09:40:48 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 09:40:48 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 32 previous similar messages Jun 22 09:42:38 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 09:42:38 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2753 previous similar messages Jun 22 09:52:44 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 09:52:44 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2889 previous similar messages Jun 22 09:56:02 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 09:56:02 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jun 22 10:02:46 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 10:02:46 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3156 previous similar messages Jun 22 10:06:37 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 10:06:37 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 11 previous similar messages Jun 22 10:12:48 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 10:12:48 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2405 previous similar messages Jun 22 10:16:47 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 10:16:47 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 224 previous similar messages Jun 22 10:22:54 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 10:22:54 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1604 previous similar messages Jun 22 10:26:49 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 10:26:49 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 238 previous similar messages Jun 22 10:33:36 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 10:33:36 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 292 previous similar messages Jun 22 10:36:55 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 10:36:55 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 237 previous similar messages Jun 22 10:44:38 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 10:44:38 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 396 previous similar messages Jun 22 10:46:55 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 10:46:55 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 257 previous similar messages Jun 22 10:55:30 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 10:55:30 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 267 previous similar messages Jun 22 10:56:58 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 10:56:58 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 217 previous similar messages Jun 22 11:05:37 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 11:05:37 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 313 previous similar messages Jun 22 11:06:58 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 11:06:58 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 246 previous similar messages Jun 22 11:16:07 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 11:16:07 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 475 previous similar messages Jun 22 11:16:58 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 45056 GRANT, real grant 0 Jun 22 11:16:58 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 216 previous similar messages Jun 22 11:26:27 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 11:26:27 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 372 previous similar messages Jun 22 11:26:59 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 11:26:59 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 244 previous similar messages Jun 22 11:36:59 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 11:36:59 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 220 previous similar messages Jun 22 11:37:03 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 11:37:03 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 229 previous similar messages Jun 22 11:47:08 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 11:47:08 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 249 previous similar messages Jun 22 11:51:18 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 11:51:18 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 319 previous similar messages Jun 22 11:57:22 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 69632 GRANT, real grant 0 Jun 22 11:57:22 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 220 previous similar messages Jun 22 12:02:26 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 12:02:26 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 193 previous similar messages Jun 22 12:07:25 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 12:07:25 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 228 previous similar messages Jun 22 12:08:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 2ca8c1ab-ca57-7d24-398b-275ee2691945 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4537aa7800, cur 1561230506 expire 1561230356 last 1561230279 Jun 22 12:08:26 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 22 12:08:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 2ca8c1ab-ca57-7d24-398b-275ee2691945 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2520a3c400, cur 1561230508 expire 1561230358 last 1561230281 Jun 22 12:08:28 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 22 12:17:29 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 12:17:29 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 214 previous similar messages Jun 22 12:27:32 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 12:27:32 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 247 previous similar messages Jun 22 12:32:52 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 12:32:52 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 31 previous similar messages Jun 22 12:34:17 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 12:34:17 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 32 previous similar messages Jun 22 12:37:02 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 12:37:02 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 21 previous similar messages Jun 22 12:37:39 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 12:37:39 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 217 previous similar messages Jun 22 12:43:40 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 12:43:40 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 141 previous similar messages Jun 22 12:47:40 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 12:47:40 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 249 previous similar messages Jun 22 12:54:00 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 12:54:00 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 49 previous similar messages Jun 22 12:57:43 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 22 12:57:43 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 423 previous similar messages Jun 22 13:07:47 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 22 13:07:47 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1294 previous similar messages Jun 22 13:09:41 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 13:09:41 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 31 previous similar messages Jun 22 13:17:50 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 22 13:17:50 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1284 previous similar messages Jun 22 13:27:51 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 13:27:51 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1348 previous similar messages Jun 22 13:28:18 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 13:28:18 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages Jun 22 13:31:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e6c09851-8594-e724-7da8-570118535052 (at 10.9.107.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2533e57400, cur 1561235462 expire 1561235312 last 1561235235 Jun 22 13:37:51 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 13:37:51 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 611 previous similar messages Jun 22 13:43:06 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 13:47:51 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 13:47:51 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 448 previous similar messages Jun 22 13:50:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 286d4aef-dd39-033a-885a-1b2f68dad8ee (at 10.9.112.16@o2ib4) Jun 22 13:50:28 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 22 13:50:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to f60c199d-7611-7247-14ce-916a8ab83213 (at 10.9.112.13@o2ib4) Jun 22 13:50:33 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 22 13:53:09 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 13:53:09 fir-md1-s1 kernel: Lustre: 20541:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 169 previous similar messages Jun 22 13:53:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1cdcf44c-092e-67dd-29a2-3cb7e9bc7e29 (at 10.8.15.6@o2ib6) Jun 22 13:53:58 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 22 13:57:52 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 13:57:52 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 446 previous similar messages Jun 22 13:59:39 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c579ffa9-959a-5f2e-006d-9d0dfdb5fa5a (at 10.8.17.26@o2ib6) Jun 22 13:59:39 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 22 14:00:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to dd99d9ee-4aca-6a76-941f-529d29521420 (at 10.8.2.28@o2ib6) Jun 22 14:00:13 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 22 14:01:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to fc2d5c6b-10cd-8ca7-0b9f-2fba82f0b956 (at 10.8.10.27@o2ib6) Jun 22 14:01:21 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 22 14:04:29 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 14:04:29 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 287 previous similar messages Jun 22 14:07:53 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 14:07:53 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 425 previous similar messages Jun 22 14:14:29 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 14:14:29 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1720 previous similar messages Jun 22 14:17:53 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 14:17:53 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 401 previous similar messages Jun 22 14:27:16 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 14:27:16 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 595 previous similar messages Jun 22 14:27:54 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 14:27:54 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 409 previous similar messages Jun 22 14:37:54 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 14:37:54 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 22 14:39:20 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 14:39:20 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 127 previous similar messages Jun 22 14:47:55 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 14:47:55 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 22 14:51:13 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 14:51:13 fir-md1-s1 kernel: Lustre: 21419:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 22 14:57:56 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 14:57:56 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 409 previous similar messages Jun 22 15:01:27 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 15:01:27 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages Jun 22 15:07:57 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 15:07:57 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 397 previous similar messages Jun 22 15:11:27 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 15:11:27 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 50 previous similar messages Jun 22 15:17:57 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 15:17:57 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 22 15:21:29 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 15:21:29 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 62 previous similar messages Jun 22 15:27:58 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 15:27:58 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 405 previous similar messages Jun 22 15:31:30 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 15:31:30 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 743 previous similar messages Jun 22 15:37:58 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 15:37:58 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 22 15:41:31 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 15:41:31 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1274 previous similar messages Jun 22 15:47:58 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 15:47:58 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 415 previous similar messages Jun 22 15:51:36 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 15:51:36 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1693 previous similar messages Jun 22 15:57:59 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 15:57:59 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 410 previous similar messages Jun 22 16:03:01 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 16:03:01 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 8 previous similar messages Jun 22 16:08:00 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 16:08:00 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 419 previous similar messages Jun 22 16:13:10 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 16:13:10 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages Jun 22 16:18:01 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 16:18:01 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 407 previous similar messages Jun 22 16:23:11 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 16:23:11 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 251 previous similar messages Jun 22 16:28:01 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 16:28:01 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 420 previous similar messages Jun 22 16:33:12 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 16:33:12 fir-md1-s1 kernel: Lustre: 21073:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1059 previous similar messages Jun 22 16:38:02 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 16:38:02 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 424 previous similar messages Jun 22 16:43:12 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 16:43:12 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2159 previous similar messages Jun 22 16:48:02 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 16:48:02 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 420 previous similar messages Jun 22 16:53:12 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 16:53:12 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14293 previous similar messages Jun 22 16:58:04 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 16:58:04 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 429 previous similar messages Jun 22 17:04:03 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 17:04:03 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 16274 previous similar messages Jun 22 17:08:05 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 17:08:05 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 406 previous similar messages Jun 22 17:14:03 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 17:14:03 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1280 previous similar messages Jun 22 17:18:05 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 17:18:05 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 22 17:24:05 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 17:24:05 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 176 previous similar messages Jun 22 17:28:06 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 17:28:06 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 407 previous similar messages Jun 22 17:34:22 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 17:34:22 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 86 previous similar messages Jun 22 17:38:06 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 17:38:06 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 22 17:44:26 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 17:44:26 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 382 previous similar messages Jun 22 17:48:07 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 17:48:07 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 409 previous similar messages Jun 22 17:54:26 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 17:54:26 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1754 previous similar messages Jun 22 17:58:07 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 17:58:07 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 406 previous similar messages Jun 22 18:08:08 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 18:08:08 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 408 previous similar messages Jun 22 18:14:04 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 18:14:04 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 22 18:18:09 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 18:18:09 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 406 previous similar messages Jun 22 18:25:53 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 18:25:53 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages Jun 22 18:28:10 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 18:28:10 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 431 previous similar messages Jun 22 18:38:11 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 18:38:11 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 405 previous similar messages Jun 22 18:42:48 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 18:42:48 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages Jun 22 18:48:12 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 18:48:12 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 22 18:54:11 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 18:54:11 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages Jun 22 18:58:13 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 18:58:13 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 404 previous similar messages Jun 22 19:08:14 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 19:08:14 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 521 previous similar messages Jun 22 19:09:59 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 19:09:59 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 22 19:18:16 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 19:18:16 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 394 previous similar messages Jun 22 19:21:16 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 19:21:16 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages Jun 22 19:28:16 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 19:28:16 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 380 previous similar messages Jun 22 19:35:43 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 19:35:43 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 54 previous similar messages Jun 22 19:38:16 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 19:38:16 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 391 previous similar messages Jun 22 19:48:18 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 19:48:18 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 384 previous similar messages Jun 22 19:55:27 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 19:55:27 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 30 previous similar messages Jun 22 19:58:18 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 19:58:18 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 379 previous similar messages Jun 22 20:08:07 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 20:08:07 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11 previous similar messages Jun 22 20:08:19 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 20:08:19 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jun 22 20:18:19 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 20:18:19 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 22 20:28:20 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 20:28:20 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 382 previous similar messages Jun 22 20:29:28 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 20:29:28 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages Jun 22 20:33:45 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 20:33:45 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 2 previous similar messages Jun 22 20:38:21 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 20:38:21 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 391 previous similar messages Jun 22 20:42:16 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 20:48:21 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 20:48:21 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 381 previous similar messages Jun 22 20:58:22 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 20:58:22 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 380 previous similar messages Jun 22 20:58:27 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 20:58:27 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 5 previous similar messages Jun 22 20:59:34 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 20:59:34 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 170 previous similar messages Jun 22 21:00:49 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 21:00:49 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3728 previous similar messages Jun 22 21:03:47 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 21:03:47 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3997 previous similar messages Jun 22 21:08:23 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 21:08:23 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 390 previous similar messages Jun 22 21:08:48 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 21:08:48 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 28782 previous similar messages Jun 22 21:18:24 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 21:18:24 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 387 previous similar messages Jun 22 21:19:40 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 21:19:40 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 782 previous similar messages Jun 22 21:28:24 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 21:28:24 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 376 previous similar messages Jun 22 21:38:25 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 21:38:25 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 396 previous similar messages Jun 22 21:38:57 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 21:38:57 fir-md1-s1 kernel: Lustre: 21312:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 312 previous similar messages Jun 22 21:48:26 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 21:48:26 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 22 21:49:46 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 21:49:46 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 6 previous similar messages Jun 22 21:58:27 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 21:58:27 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 404 previous similar messages Jun 22 22:00:12 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 22:00:12 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 20 previous similar messages Jun 22 22:08:27 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 22:08:27 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 400 previous similar messages Jun 22 22:10:12 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 22:10:12 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages Jun 22 22:18:29 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 22:18:29 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 382 previous similar messages Jun 22 22:28:30 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 22:28:30 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 384 previous similar messages Jun 22 22:38:31 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 22:38:31 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 404 previous similar messages Jun 22 22:39:18 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 22:39:18 fir-md1-s1 kernel: Lustre: 27321:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages Jun 22 22:44:23 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 22:44:23 fir-md1-s1 kernel: Lustre: 25681:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages Jun 22 22:46:53 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 22:46:53 fir-md1-s1 kernel: Lustre: 21673:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 47930 previous similar messages Jun 22 22:48:31 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 22:48:31 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 418 previous similar messages Jun 22 22:52:00 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 22:52:00 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 56761 previous similar messages Jun 22 22:58:32 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 22:58:32 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 425 previous similar messages Jun 22 23:02:03 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 23:02:03 fir-md1-s1 kernel: Lustre: 21369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 11355 previous similar messages Jun 22 23:08:33 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 23:08:33 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 423 previous similar messages Jun 22 23:12:30 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 23:12:30 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 35 previous similar messages Jun 22 23:18:34 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 23:18:34 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 421 previous similar messages Jun 22 23:28:35 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 23:28:35 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 419 previous similar messages Jun 22 23:37:06 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 23:37:06 fir-md1-s1 kernel: Lustre: 25678:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages Jun 22 23:38:35 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 23:38:35 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 413 previous similar messages Jun 22 23:40:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 8fc07523-e22f-bfb9-0ffa-4aa1d872317e (at 10.8.7.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f451891f400, cur 1561272050 expire 1561271900 last 1561271823 Jun 22 23:40:50 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jun 22 23:42:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ea1cdf3e-c1a9-c826-73a8-fd54bacafbe5 (at 10.8.7.4@o2ib6) Jun 22 23:42:34 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jun 22 23:45:08 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 23:45:08 fir-md1-s1 kernel: Lustre: 20459:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 22 23:47:56 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 23:47:56 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 80 previous similar messages Jun 22 23:48:36 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 23:48:36 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 420 previous similar messages Jun 22 23:53:03 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 22 23:53:03 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 194 previous similar messages Jun 22 23:58:37 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 22 23:58:37 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 414 previous similar messages Jun 23 00:03:34 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 00:03:34 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 198 previous similar messages Jun 23 00:08:38 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 00:08:38 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 420 previous similar messages Jun 23 00:14:08 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 00:14:08 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 377 previous similar messages Jun 23 00:18:39 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 00:18:39 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 427 previous similar messages Jun 23 00:24:17 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 00:24:17 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 193 previous similar messages Jun 23 00:28:41 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 00:28:41 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 416 previous similar messages Jun 23 00:35:02 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 00:35:02 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 292 previous similar messages Jun 23 00:38:42 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 00:38:42 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 415 previous similar messages Jun 23 00:45:05 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 00:45:05 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 96 previous similar messages Jun 23 00:48:43 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 00:48:43 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 416 previous similar messages Jun 23 00:55:14 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 00:55:14 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 599 previous similar messages Jun 23 00:58:44 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 00:58:44 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2150 previous similar messages Jun 23 01:05:50 fir-md1-s1 kernel: Lustre: 10308:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 01:05:50 fir-md1-s1 kernel: Lustre: 10308:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1487 previous similar messages Jun 23 01:08:45 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 01:08:45 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 802 previous similar messages Jun 23 01:15:52 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 01:15:52 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 77 previous similar messages Jun 23 01:18:46 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 01:18:46 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 456 previous similar messages Jun 23 01:27:43 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 01:27:43 fir-md1-s1 kernel: Lustre: 27319:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 15 previous similar messages Jun 23 01:28:46 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 01:28:46 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1111 previous similar messages Jun 23 01:38:48 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 01:38:48 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1125 previous similar messages Jun 23 01:48:48 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 23 01:48:48 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1396 previous similar messages Jun 23 01:52:53 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 01:52:53 fir-md1-s1 kernel: Lustre: 25676:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 171 previous similar messages Jun 23 01:54:27 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 01:54:27 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages Jun 23 01:58:49 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 01:58:49 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1178 previous similar messages Jun 23 02:08:49 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 02:08:49 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1302 previous similar messages Jun 23 02:10:03 fir-md1-s1 kernel: Lustre: 20738:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 02:18:51 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 02:18:51 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1208 previous similar messages Jun 23 02:19:45 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 02:20:41 fir-md1-s1 kernel: Lustre: 10308:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 02:20:41 fir-md1-s1 kernel: Lustre: 10308:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 21 previous similar messages Jun 23 02:21:57 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 02:21:57 fir-md1-s1 kernel: Lustre: 21410:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 193 previous similar messages Jun 23 02:24:31 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 02:24:31 fir-md1-s1 kernel: Lustre: 21421:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 138 previous similar messages Jun 23 02:28:51 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 23 02:28:51 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2402 previous similar messages Jun 23 02:36:30 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 02:36:30 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 131 previous similar messages Jun 23 02:38:52 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 02:38:52 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1179 previous similar messages Jun 23 02:46:44 fir-md1-s1 kernel: Lustre: 10307:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 02:46:44 fir-md1-s1 kernel: Lustre: 10307:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1140 previous similar messages Jun 23 02:48:53 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 02:48:53 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1267 previous similar messages Jun 23 02:58:54 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 02:58:54 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 775 previous similar messages Jun 23 03:08:55 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 03:08:55 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 816 previous similar messages Jun 23 03:17:08 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 03:17:08 fir-md1-s1 kernel: Lustre: 21368:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 838 previous similar messages Jun 23 03:18:55 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 03:18:55 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1679 previous similar messages Jun 23 03:19:17 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 03:19:17 fir-md1-s1 kernel: Lustre: 21416:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 37 previous similar messages Jun 23 03:22:05 fir-md1-s1 kernel: Lustre: 10196:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 03:22:05 fir-md1-s1 kernel: Lustre: 10196:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages Jun 23 03:27:17 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 03:27:17 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 142 previous similar messages Jun 23 03:28:57 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 03:28:57 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1219 previous similar messages Jun 23 03:38:57 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 03:38:57 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1149 previous similar messages Jun 23 03:48:58 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 61440 GRANT, real grant 0 Jun 23 03:48:58 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1576 previous similar messages Jun 23 03:58:58 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 03:58:58 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1228 previous similar messages Jun 23 04:09:00 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 04:09:00 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1332 previous similar messages Jun 23 04:19:00 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 04:19:00 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3231 previous similar messages Jun 23 04:29:01 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 04:29:01 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 905 previous similar messages Jun 23 04:39:02 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 04:39:02 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1154 previous similar messages Jun 23 04:49:02 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 04:49:02 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1147 previous similar messages Jun 23 04:59:03 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 04:59:03 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1436 previous similar messages Jun 23 05:09:03 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 147456 GRANT, real grant 0 Jun 23 05:09:03 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1435 previous similar messages Jun 23 05:19:03 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 61440 GRANT, real grant 0 Jun 23 05:19:03 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1395 previous similar messages Jun 23 05:29:04 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 05:29:04 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1483 previous similar messages Jun 23 05:39:05 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 05:39:05 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1391 previous similar messages Jun 23 05:49:06 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 05:49:06 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1465 previous similar messages Jun 23 05:59:07 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 05:59:07 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1578 previous similar messages Jun 23 06:09:08 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 06:09:08 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1494 previous similar messages Jun 23 06:19:09 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 06:19:09 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1266 previous similar messages Jun 23 06:29:09 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 23 06:29:09 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1203 previous similar messages Jun 23 06:39:09 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 23 06:39:09 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1386 previous similar messages Jun 23 06:49:10 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 06:49:10 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1413 previous similar messages Jun 23 06:59:12 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 06:59:12 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1391 previous similar messages Jun 23 07:09:12 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 23 07:09:12 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1183 previous similar messages Jun 23 07:19:12 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 07:19:12 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1037 previous similar messages Jun 23 07:29:13 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 07:29:13 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 931 previous similar messages Jun 23 07:39:13 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 23 07:39:13 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1132 previous similar messages Jun 23 07:49:13 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 07:49:13 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1108 previous similar messages Jun 23 07:59:14 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 07:59:14 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 999 previous similar messages Jun 23 08:09:15 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 08:09:15 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 932 previous similar messages Jun 23 08:19:15 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 08:19:15 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1250 previous similar messages Jun 23 08:29:16 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 08:29:16 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1334 previous similar messages Jun 23 08:39:16 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 98304 GRANT, real grant 0 Jun 23 08:39:16 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 30331 previous similar messages Jun 23 08:49:16 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 08:49:16 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 30305 previous similar messages Jun 23 08:56:31 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 23 08:56:31 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 100 previous similar messages Jun 23 08:59:17 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 08:59:17 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1472 previous similar messages Jun 23 09:08:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f15da91-4546-507e-8c99-9e08b5e219a4 (at 10.8.15.10@o2ib6) Jun 23 09:08:33 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 23 09:09:17 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 09:09:17 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 400 previous similar messages Jun 23 09:19:17 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 09:19:17 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 418 previous similar messages Jun 23 09:23:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f15da91-4546-507e-8c99-9e08b5e219a4 (at 10.8.15.10@o2ib6) Jun 23 09:23:47 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 23 09:29:17 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 09:29:17 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 422 previous similar messages Jun 23 09:39:19 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 09:39:19 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 434 previous similar messages Jun 23 09:49:19 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 09:49:19 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 406 previous similar messages Jun 23 09:59:20 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 09:59:20 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 412 previous similar messages Jun 23 10:09:22 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 10:09:22 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 409 previous similar messages Jun 23 10:19:22 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 10:19:22 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 693 previous similar messages Jun 23 10:29:24 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 10:29:24 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 834 previous similar messages Jun 23 10:31:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client f1559460-8fda-b79d-be15-a1d7dda11872 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f45181f9c00, cur 1561311114 expire 1561310964 last 1561310887 Jun 23 10:31:54 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 23 10:33:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d8cc7b58-ee01-5501-ca65-c659f4724147 (at 10.9.106.54@o2ib4) Jun 23 10:33:35 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 23 10:39:24 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 23 10:39:24 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 653 previous similar messages Jun 23 10:49:25 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 10:49:25 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 645 previous similar messages Jun 23 10:59:26 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 10:59:26 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 638 previous similar messages Jun 23 11:09:27 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 11:09:27 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 600 previous similar messages Jun 23 11:19:27 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 23 11:19:27 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 629 previous similar messages Jun 23 11:22:54 fir-md1-s1 kernel: Lustre: 20458:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561314167/real 1561314167] req@ffff8f0c20f08300 x1636711856945808/t0(0) o106->fir-MDT0002@10.9.106.54@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1561314174 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 23 11:23:02 fir-md1-s1 kernel: Lustre: 21673:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0abb318000 x1634936627028896/t0(0) o101->bd073587-8042-ffd0-09f1-ff79e8722875@10.9.0.63@o2ib4:7/0 lens 480/568 e 1 to 0 dl 1561314187 ref 2 fl Interpret:/0/0 rc 0/0 Jun 23 11:23:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) reconnecting Jun 23 11:23:08 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 23 11:23:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 23 11:23:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 23 11:23:15 fir-md1-s1 kernel: Lustre: 20458:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561314188/real 1561314188] req@ffff8f0c20f08300 x1636711856945808/t0(0) o106->fir-MDT0002@10.9.106.54@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1561314195 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 23 11:23:15 fir-md1-s1 kernel: Lustre: 20458:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 23 11:23:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) reconnecting Jun 23 11:23:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 23 11:23:50 fir-md1-s1 kernel: Lustre: 20458:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561314223/real 1561314223] req@ffff8f0c20f08300 x1636711856945808/t0(0) o106->fir-MDT0002@10.9.106.54@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1561314230 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 23 11:23:50 fir-md1-s1 kernel: Lustre: 20458:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 23 11:23:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) reconnecting Jun 23 11:23:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 23 11:24:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) reconnecting Jun 23 11:24:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 23 11:24:31 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1fda306300 x1631561955722480/t0(0) o101->ebb0ff39-b00e-6e1a-c25b-64754a77a1b9@10.8.0.82@o2ib6:6/0 lens 576/3264 e 1 to 0 dl 1561314276 ref 2 fl Interpret:/0/0 rc 0/0 Jun 23 11:24:32 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1d637b7b00 x1631562793996352/t0(0) o101->d594a152-d993-c755-50bf-0f3b806ddc60@10.9.107.22@o2ib4:7/0 lens 576/0 e 1 to 0 dl 1561314277 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 23 11:24:32 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 523 previous similar messages Jun 23 11:24:34 fir-md1-s1 kernel: Lustre: 20462:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2085b89800 x1631567984096800/t0(0) o101->442dd3b5-503d-fa23-0886-f83a3c7ec479@10.8.18.5@o2ib6:9/0 lens 576/0 e 1 to 0 dl 1561314279 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 23 11:24:34 fir-md1-s1 kernel: Lustre: 20462:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 142 previous similar messages Jun 23 11:24:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) reconnecting Jun 23 11:24:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 23 11:24:38 fir-md1-s1 kernel: Lustre: 20462:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1fcda53600 x1634929795870016/t0(0) o101->749699ee-a0f2-6ab2-f022-71007184e2c9@10.8.8.23@o2ib6:13/0 lens 576/0 e 1 to 0 dl 1561314283 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 23 11:24:38 fir-md1-s1 kernel: Lustre: 20462:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 185 previous similar messages Jun 23 11:24:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b743407c-1f2a-22f5-529c-bf172a166e4e (at 10.8.2.20@o2ib6) Jun 23 11:24:42 fir-md1-s1 kernel: Lustre: Skipped 370 previous similar messages Jun 23 11:24:46 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f19acb25100 x1631582681370096/t0(0) o101->aba5d4eb-e07c-9b0f-6ab5-7f97caf38a26@10.8.16.4@o2ib6:21/0 lens 576/0 e 1 to 0 dl 1561314291 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 23 11:24:46 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 224 previous similar messages Jun 23 11:24:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client de259a64-2100-eb0d-e7c9-3532a08afec2 (at 10.9.102.41@o2ib4) reconnecting Jun 23 11:24:51 fir-md1-s1 kernel: Lustre: Skipped 523 previous similar messages Jun 23 11:24:58 fir-md1-s1 kernel: Lustre: 25677:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561314291/real 1561314291] req@ffff8f450387ec00 x1636711857283520/t0(0) o104->fir-MDT0000@10.9.106.54@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1561314298 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 23 11:24:58 fir-md1-s1 kernel: Lustre: 25677:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jun 23 11:24:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to f54ca57d-f21f-fc73-ad63-df7922956fa9 (at 10.9.102.40@o2ib4) Jun 23 11:24:58 fir-md1-s1 kernel: Lustre: Skipped 419 previous similar messages Jun 23 11:25:02 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f18d08b4200 x1631538541031232/t0(0) o101->c3098872-1c7c-63b2-cf3c-a9a145f04126@10.8.18.31@o2ib6:7/0 lens 576/0 e 1 to 0 dl 1561314307 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 23 11:25:02 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 784 previous similar messages Jun 23 11:25:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client c534882d-6030-1b8a-8c54-b433ef117432 (at 10.9.108.56@o2ib4) reconnecting Jun 23 11:25:23 fir-md1-s1 kernel: Lustre: Skipped 1151 previous similar messages Jun 23 11:25:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9bc02376-d562-fb9e-7cb7-2dc944d1678e (at 10.9.101.67@o2ib4) Jun 23 11:25:30 fir-md1-s1 kernel: Lustre: Skipped 1127 previous similar messages Jun 23 11:25:34 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1837d3f800 x1635373772455680/t0(0) o101->6d0f4c77-c27b-6d80-d629-873de917b74e@10.8.0.66@o2ib6:9/0 lens 576/0 e 0 to 0 dl 1561314339 ref 2 fl New:/2/ffffffff rc 0/-1 Jun 23 11:25:34 fir-md1-s1 kernel: Lustre: 21447:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1266 previous similar messages Jun 23 11:25:46 fir-md1-s1 kernel: LustreError: 21461:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561314256, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1b586a5100/0x5d9ee62174100794 lrc: 3/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 1156 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21461 timeout: 0 lvb_type: 0 Jun 23 11:25:46 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561314346.24576 Jun 23 11:25:46 fir-md1-s1 kernel: LustreError: 21461:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 194 previous similar messages Jun 23 11:25:46 fir-md1-s1 kernel: LustreError: 23609:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561314256, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f06a6201d40/0x5d9ee62174101c78 lrc: 3/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 1156 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23609 timeout: 0 lvb_type: 0 Jun 23 11:25:46 fir-md1-s1 kernel: LustreError: 23609:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 70 previous similar messages Jun 23 11:25:47 fir-md1-s1 kernel: LustreError: 23679:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561314257, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f119a36b840/0x5d9ee62174102022 lrc: 3/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 1156 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23679 timeout: 0 lvb_type: 0 Jun 23 11:25:47 fir-md1-s1 kernel: LustreError: 23679:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 61 previous similar messages Jun 23 11:25:49 fir-md1-s1 kernel: LustreError: 23729:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561314259, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f45180018c0/0x5d9ee621741022d0 lrc: 3/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 1156 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23729 timeout: 0 lvb_type: 0 Jun 23 11:25:49 fir-md1-s1 kernel: LustreError: 23729:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 51 previous similar messages Jun 23 11:26:08 fir-md1-s1 kernel: LNet: Service thread pid 20458 was inactive for 200.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 23 11:26:08 fir-md1-s1 kernel: Pid: 20458, comm: mdt00_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 23 11:26:08 fir-md1-s1 kernel: Call Trace: Jun 23 11:26:08 fir-md1-s1 kernel: [] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] mdt_do_glimpse+0x1e9/0x4c0 [mdt] Jun 23 11:26:08 fir-md1-s1 kernel: [] mdt_glimpse_enqueue+0x3d3/0x4f0 [mdt] Jun 23 11:26:08 fir-md1-s1 kernel: [] mdt_intent_glimpse+0x1f/0x30 [mdt] Jun 23 11:26:08 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jun 23 11:26:08 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 23 11:26:08 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 23 11:26:08 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 23 11:26:08 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 23 11:26:08 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561314368.20458 Jun 23 11:26:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client a63e3144-5861-13b0-6a48-7b4c39aca713 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0fb0d13000, cur 1561314380 expire 1561314230 last 1561314153 Jun 23 11:26:20 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 23 11:26:20 fir-md1-s1 kernel: Lustre: 23761:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:98s); client may timeout. req@ffff8f2baeefe600 x1631537339309024/t0(0) o101->f295817f-4700-452c-6407-60dfd6afbd18@10.9.104.4@o2ib4:12/0 lens 576/0 e 1 to 0 dl 1561314282 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jun 23 11:26:20 fir-md1-s1 kernel: LustreError: 25680:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.109.3@o2ib4: deadline 30:1s ago req@ffff8f3519ff4850 x1631615048905472/t0(0) o101->09300796-1183-3575-4e70-90c873be0aeb@10.9.109.3@o2ib4:19/0 lens 576/0 e 0 to 0 dl 1561314379 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jun 23 11:26:20 fir-md1-s1 kernel: LNet: Service thread pid 20458 completed after 212.35s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 23 11:26:20 fir-md1-s1 kernel: Lustre: 23761:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 4721 previous similar messages Jun 23 11:26:20 fir-md1-s1 kernel: LustreError: 22280:0:(tgt_handler.c:644:process_req_last_xid()) @@@ Unexpected xid 5cf4e95db6620 vs. last_xid 5cf4e95db677f req@ffff8f178441da00 x1635311312135712/t0(0) o101->c33dfd3e-93e2-b1e4-c92b-6be01740e2e1@10.9.115.7@o2ib4:20/0 lens 576/0 e 0 to 0 dl 1561314410 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jun 23 11:26:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 397e53ea-489f-22f1-95c4-27ab82ab5709 (at 10.9.102.43@o2ib4) reconnecting Jun 23 11:26:28 fir-md1-s1 kernel: Lustre: Skipped 1985 previous similar messages Jun 23 11:26:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.108.22@o2ib4) Jun 23 11:26:35 fir-md1-s1 kernel: Lustre: Skipped 1763 previous similar messages Jun 23 11:28:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d8cc7b58-ee01-5501-ca65-c659f4724147 (at 10.9.106.54@o2ib4) Jun 23 11:28:58 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jun 23 11:29:28 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 69632 GRANT, real grant 0 Jun 23 11:29:28 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 481 previous similar messages Jun 23 11:39:28 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 11:39:28 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 651 previous similar messages Jun 23 11:49:28 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 23 11:49:28 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 615 previous similar messages Jun 23 11:59:29 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 11:59:29 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 636 previous similar messages Jun 23 12:09:30 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 12:09:30 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 603 previous similar messages Jun 23 12:19:31 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 12:19:31 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 615 previous similar messages Jun 23 12:29:31 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 98304 GRANT, real grant 0 Jun 23 12:29:31 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 630 previous similar messages Jun 23 12:39:32 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 90112 GRANT, real grant 0 Jun 23 12:39:32 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 614 previous similar messages Jun 23 12:49:33 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 12:49:33 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 628 previous similar messages Jun 23 12:59:34 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 12:59:34 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 618 previous similar messages Jun 23 13:09:34 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 13:09:34 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1331 previous similar messages Jun 23 13:19:35 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 13:19:35 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1627 previous similar messages Jun 23 13:29:36 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 13:29:36 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1629 previous similar messages Jun 23 13:39:37 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 13:39:37 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1386 previous similar messages Jun 23 13:49:37 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 13:49:37 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 394 previous similar messages Jun 23 13:59:38 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 13:59:38 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 398 previous similar messages Jun 23 14:09:38 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 14:09:38 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 386 previous similar messages Jun 23 14:19:39 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 14:19:39 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 384 previous similar messages Jun 23 14:29:40 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 14:29:40 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 23 14:39:41 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 14:39:41 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 383 previous similar messages Jun 23 14:49:41 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 14:49:41 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 411 previous similar messages Jun 23 14:59:41 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 14:59:41 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 417 previous similar messages Jun 23 15:09:42 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 15:09:42 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 423 previous similar messages Jun 23 15:19:43 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 15:19:43 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 417 previous similar messages Jun 23 15:29:44 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 15:29:44 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1694 previous similar messages Jun 23 15:39:44 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 15:39:44 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 23 15:49:44 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 15:49:44 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 610 previous similar messages Jun 23 15:59:46 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 15:59:46 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 899 previous similar messages Jun 23 16:09:46 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 23 16:09:46 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 815 previous similar messages Jun 23 16:19:47 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 16:19:47 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 898 previous similar messages Jun 23 16:29:47 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 16:29:47 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 725 previous similar messages Jun 23 16:39:47 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 16:39:47 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1026 previous similar messages Jun 23 16:49:48 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 16:49:48 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1028 previous similar messages Jun 23 16:59:50 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 16:59:50 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 618 previous similar messages Jun 23 17:09:50 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 17:09:50 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 381 previous similar messages Jun 23 17:19:51 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 17:19:51 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 391 previous similar messages Jun 23 17:29:52 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 17:29:52 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 379 previous similar messages Jun 23 17:39:53 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 17:39:53 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 386 previous similar messages Jun 23 17:49:53 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 17:49:53 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 391 previous similar messages Jun 23 17:59:55 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 17:59:55 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 378 previous similar messages Jun 23 18:09:55 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 18:09:55 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 381 previous similar messages Jun 23 18:19:56 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 18:19:56 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 393 previous similar messages Jun 23 18:29:57 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 18:29:57 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 396 previous similar messages Jun 23 18:39:57 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 18:39:57 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 409 previous similar messages Jun 23 18:49:57 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 18:49:57 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 367 previous similar messages Jun 23 18:59:58 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 18:59:58 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 405 previous similar messages Jun 23 19:09:58 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 19:09:58 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 411 previous similar messages Jun 23 19:19:59 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 19:19:59 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 413 previous similar messages Jun 23 19:30:00 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 19:30:00 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 410 previous similar messages Jun 23 19:40:00 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 19:40:00 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 415 previous similar messages Jun 23 19:50:01 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 19:50:01 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 406 previous similar messages Jun 23 20:00:02 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 20:00:02 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 412 previous similar messages Jun 23 20:10:02 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 20:10:02 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 415 previous similar messages Jun 23 20:20:03 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 20:20:03 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 415 previous similar messages Jun 23 20:30:04 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 20:30:04 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 414 previous similar messages Jun 23 20:40:05 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 20:40:05 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 417 previous similar messages Jun 23 20:50:05 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 20:50:05 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 420 previous similar messages Jun 23 21:00:09 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 21:00:09 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 420 previous similar messages Jun 23 21:10:10 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 21:10:10 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 418 previous similar messages Jun 23 21:20:11 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 21:20:11 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 398 previous similar messages Jun 23 21:30:11 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 21:30:11 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 431 previous similar messages Jun 23 21:40:13 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 21:40:13 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 409 previous similar messages Jun 23 21:50:14 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 21:50:14 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 413 previous similar messages Jun 23 22:00:14 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 22:00:14 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 410 previous similar messages Jun 23 22:10:15 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 22:10:15 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jun 23 22:20:17 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 22:20:17 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 444 previous similar messages Jun 23 22:30:18 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 22:30:18 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 416 previous similar messages Jun 23 22:40:19 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 22:40:19 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 428 previous similar messages Jun 23 22:50:19 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 22:50:19 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 421 previous similar messages Jun 23 23:00:20 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 23:00:20 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 444 previous similar messages Jun 23 23:10:20 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 23:10:20 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 434 previous similar messages Jun 23 23:20:20 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 23:20:20 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 427 previous similar messages Jun 23 23:30:20 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 23:30:20 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 452 previous similar messages Jun 23 23:40:20 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 23:40:20 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 442 previous similar messages Jun 23 23:50:21 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 23 23:50:21 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 444 previous similar messages Jun 24 00:00:22 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 00:00:22 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 526 previous similar messages Jun 24 00:10:22 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 77824 GRANT, real grant 0 Jun 24 00:10:22 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1469 previous similar messages Jun 24 00:20:23 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 00:20:23 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1294 previous similar messages Jun 24 00:30:24 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 00:30:24 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 452 previous similar messages Jun 24 00:40:24 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 00:40:24 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 423 previous similar messages Jun 24 00:50:25 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 00:50:25 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 875 previous similar messages Jun 24 00:56:28 fir-md1-s1 kernel: Lustre: 23632:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jun 24 01:00:26 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 98304 GRANT, real grant 0 Jun 24 01:00:26 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3524 previous similar messages Jun 24 01:10:27 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 01:10:27 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1461 previous similar messages Jun 24 01:20:27 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 01:20:27 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1715 previous similar messages Jun 24 01:30:27 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 01:30:27 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2281 previous similar messages Jun 24 01:35:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 33dff121-95b2-ba7a-9b08-f634d4e72016 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3c81bd0000, cur 1561365341 expire 1561365191 last 1561365114 Jun 24 01:35:41 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 01:40:28 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 01:40:28 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2180 previous similar messages Jun 24 01:50:28 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 131072 GRANT, real grant 0 Jun 24 01:50:28 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2166 previous similar messages Jun 24 02:00:28 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 24 02:00:28 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2162 previous similar messages Jun 24 02:10:29 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 02:10:29 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2193 previous similar messages Jun 24 02:20:30 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 02:20:30 fir-md1-s1 kernel: LustreError: 21365:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1878 previous similar messages Jun 24 02:30:30 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 02:30:30 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3293 previous similar messages Jun 24 02:40:31 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 02:40:31 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2339 previous similar messages Jun 24 02:50:31 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 02:50:31 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2176 previous similar messages Jun 24 03:00:31 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 40960 GRANT, real grant 0 Jun 24 03:00:31 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1869 previous similar messages Jun 24 03:10:32 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 28672 GRANT, real grant 0 Jun 24 03:10:32 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1721 previous similar messages Jun 24 03:20:32 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 24 03:20:32 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2826 previous similar messages Jun 24 03:30:33 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 03:30:33 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1996 previous similar messages Jun 24 03:40:34 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 03:40:34 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2215 previous similar messages Jun 24 03:50:35 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 03:50:35 fir-md1-s1 kernel: LustreError: 27586:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2404 previous similar messages Jun 24 04:00:35 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 04:00:35 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2034 previous similar messages Jun 24 04:10:35 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 04:10:35 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1942 previous similar messages Jun 24 04:20:36 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 04:20:36 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2810 previous similar messages Jun 24 04:30:36 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 04:30:36 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2818 previous similar messages Jun 24 04:40:36 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 28672 GRANT, real grant 0 Jun 24 04:40:36 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1656 previous similar messages Jun 24 04:50:37 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 04:50:37 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2093 previous similar messages Jun 24 05:00:37 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 05:00:37 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2219 previous similar messages Jun 24 05:10:38 fir-md1-s1 kernel: LustreError: 27482:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 05:10:38 fir-md1-s1 kernel: LustreError: 27482:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2265 previous similar messages Jun 24 05:20:38 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 131072 GRANT, real grant 0 Jun 24 05:20:38 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1955 previous similar messages Jun 24 05:30:38 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 05:30:38 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2149 previous similar messages Jun 24 05:40:39 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 05:40:39 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2263 previous similar messages Jun 24 05:41:29 fir-md1-s1 kernel: Lustre: 23571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jun 24 05:41:29 fir-md1-s1 kernel: Lustre: 23571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 450 previous similar messages Jun 24 05:50:40 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 24 05:50:40 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2249 previous similar messages Jun 24 06:00:40 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 06:00:40 fir-md1-s1 kernel: LustreError: 27587:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2070 previous similar messages Jun 24 06:10:40 fir-md1-s1 kernel: LustreError: 25634:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 06:10:40 fir-md1-s1 kernel: LustreError: 25634:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1812 previous similar messages Jun 24 06:20:41 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 32768 GRANT, real grant 0 Jun 24 06:20:41 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1985 previous similar messages Jun 24 06:30:41 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 24 06:30:41 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2000 previous similar messages Jun 24 06:40:43 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli e18301fc-f860-0db4-bf24-6c606e0cc839 claims 155648 GRANT, real grant 0 Jun 24 06:40:43 fir-md1-s1 kernel: LustreError: 27582:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1698 previous similar messages Jun 24 06:50:43 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 06:50:43 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1690 previous similar messages Jun 24 07:00:43 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 07:00:43 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1704 previous similar messages Jun 24 07:10:45 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 07:10:45 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1877 previous similar messages Jun 24 07:20:45 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 07:20:45 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1884 previous similar messages Jun 24 07:30:45 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 07:30:45 fir-md1-s1 kernel: LustreError: 21453:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1910 previous similar messages Jun 24 07:40:46 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 07:40:46 fir-md1-s1 kernel: LustreError: 27604:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1957 previous similar messages Jun 24 07:50:47 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 07:50:47 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1806 previous similar messages Jun 24 08:00:48 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 08:00:48 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1704 previous similar messages Jun 24 08:10:48 fir-md1-s1 kernel: LustreError: 44044:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 08:10:48 fir-md1-s1 kernel: LustreError: 44044:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1826 previous similar messages Jun 24 08:20:48 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 08:20:48 fir-md1-s1 kernel: LustreError: 27584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1857 previous similar messages Jun 24 08:30:49 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 08:30:49 fir-md1-s1 kernel: LustreError: 27581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1991 previous similar messages Jun 24 08:40:49 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 08:40:49 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1771 previous similar messages Jun 24 08:50:50 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 08:50:50 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 31102 previous similar messages Jun 24 08:58:30 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 0d7a1f08-916e-8a37-613f-9b8d0fd14474 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3505f17c00, cur 1561391910 expire 1561391760 last 1561391683 Jun 24 08:58:30 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 08:58:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 2dbcbb3b-0ac9-659c-3a0d-f7bf6c0943e2 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1cd69e6400, cur 1561391918 expire 1561391768 last 1561391691 Jun 24 08:58:38 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 08:58:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ec76f1db-9c9b-bbe0-847f-90a9d517c8dc (at 10.8.9.8@o2ib6) Jun 24 08:58:42 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 09:00:51 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 09:00:51 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 31049 previous similar messages Jun 24 09:10:52 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 09:10:52 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2355 previous similar messages Jun 24 09:20:52 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 09:20:52 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1369 previous similar messages Jun 24 09:30:53 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 09:30:53 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1380 previous similar messages Jun 24 09:40:54 fir-md1-s1 kernel: LustreError: 46593:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 09:40:54 fir-md1-s1 kernel: LustreError: 46593:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1377 previous similar messages Jun 24 09:50:55 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 09:50:55 fir-md1-s1 kernel: LustreError: 21543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 964 previous similar messages Jun 24 10:00:56 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 10:00:56 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 924 previous similar messages Jun 24 10:10:57 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 10:10:57 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1015 previous similar messages Jun 24 10:20:58 fir-md1-s1 kernel: LustreError: 22157:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli be42b497-ab1b-8d58-3101-014aad577cfc claims 155648 GRANT, real grant 0 Jun 24 10:20:58 fir-md1-s1 kernel: LustreError: 22157:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1270 previous similar messages Jun 24 10:25:01 fir-md1-s1 kernel: Lustre: 24577:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f207fc0b600 x1635619423618592/t0(0) o101->d072205a-1b1b-636c-7696-e9d92af1edee@10.8.20.3@o2ib6:6/0 lens 480/568 e 1 to 0 dl 1561397106 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 10:25:01 fir-md1-s1 kernel: Lustre: 24577:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2876 previous similar messages Jun 24 10:25:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 806a1caf-1a24-de27-ca27-ac4ae7fd55bf (at 10.8.23.1@o2ib6) reconnecting Jun 24 10:25:07 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jun 24 10:25:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 3ccf2b17-86d6-784b-9db3-f8aabdd282e7 (at 10.8.23.1@o2ib6) Jun 24 10:25:07 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 10:25:09 fir-md1-s1 kernel: Lustre: 26253:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1a55f32d00 x1631544122849664/t0(0) o101->cec4ce3d-7421-61e4-362c-c29b7d79240a@10.8.27.10@o2ib6:14/0 lens 1768/0 e 1 to 0 dl 1561397114 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 24 10:25:09 fir-md1-s1 kernel: Lustre: 26253:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 354 previous similar messages Jun 24 10:25:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 6d3df076-afbd-3346-95f4-6badbc5617da (at 10.9.105.32@o2ib4) Jun 24 10:25:11 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jun 24 10:25:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 98ff8e84-1e9a-d223-7706-0c3e5612efc7 (at 10.8.0.82@o2ib6) Jun 24 10:25:19 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jun 24 10:25:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 24 10:25:23 fir-md1-s1 kernel: Lustre: Skipped 140 previous similar messages Jun 24 10:25:25 fir-md1-s1 kernel: Lustre: 26253:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f202ec24500 x1634072229373280/t0(0) o101->c6e3bcd8-71de-d683-20ac-e6684b91d659@10.9.108.10@o2ib4:0/0 lens 576/0 e 1 to 0 dl 1561397130 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 24 10:25:25 fir-md1-s1 kernel: Lustre: 26253:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 300 previous similar messages Jun 24 10:25:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 5ecd3339-79cd-a67e-2a5c-bb3ff2529a3c (at 10.8.27.10@o2ib6) Jun 24 10:25:36 fir-md1-s1 kernel: Lustre: Skipped 164 previous similar messages Jun 24 10:25:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 4dc6ad45-c67c-15d0-5638-611b0defe5f9 (at 10.8.16.2@o2ib6) reconnecting Jun 24 10:25:55 fir-md1-s1 kernel: Lustre: Skipped 341 previous similar messages Jun 24 10:25:57 fir-md1-s1 kernel: Lustre: 24577:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f16af016300 x1634121072321440/t0(0) o101->c1420e99-ffe3-a133-75d0-8971e96a81cc@10.9.106.36@o2ib4:2/0 lens 1768/0 e 1 to 0 dl 1561397162 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 24 10:25:57 fir-md1-s1 kernel: Lustre: 24577:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 557 previous similar messages Jun 24 10:26:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 9c31a52c-496a-b48d-6003-e6fdea2226d9 (at 10.9.104.22@o2ib4) Jun 24 10:26:08 fir-md1-s1 kernel: Lustre: Skipped 289 previous similar messages Jun 24 10:26:16 fir-md1-s1 kernel: LustreError: 97654:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561397086, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f0525ad6540/0x5d9ee622c3bc50ab lrc: 3/0,1 mode: --/PW res: [0x200029bbb:0xd:0x0].0x0 bits 0x40/0x0 rrc: 257 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 97654 timeout: 0 lvb_type: 0 Jun 24 10:26:16 fir-md1-s1 kernel: LustreError: 97654:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 132 previous similar messages Jun 24 10:26:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 12e474d9-b4d9-2c7f-2e45-e7d8f457f930 (at 10.8.16.8@o2ib6) reconnecting Jun 24 10:26:59 fir-md1-s1 kernel: Lustre: Skipped 724 previous similar messages Jun 24 10:27:01 fir-md1-s1 kernel: Lustre: 26253:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2071928f00 x1635085484194160/t0(0) o101->a2c269ef-57a9-8b99-0a4b-44a7d221d7bd@10.9.109.36@o2ib4:6/0 lens 1768/0 e 1 to 0 dl 1561397226 ref 2 fl New:/0/ffffffff rc 0/-1 Jun 24 10:27:01 fir-md1-s1 kernel: Lustre: 26253:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1474 previous similar messages Jun 24 10:27:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1733e647-dff2-c8f6-7390-5c06c673deac (at 10.9.109.31@o2ib4) Jun 24 10:27:12 fir-md1-s1 kernel: Lustre: Skipped 751 previous similar messages Jun 24 10:27:15 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.8.10.20@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f3fec641680/0x5d9ee622c3bc509d lrc: 3/0,0 mode: PW/PW res: [0x200029bbb:0xd:0x0].0x0 bits 0x40/0x0 rrc: 257 type: IBT flags: 0x60200400000020 nid: 10.8.10.20@o2ib6 remote: 0xc48f9d87344ea8bd expref: 85 pid: 24577 timeout: 512295 lvb_type: 0 Jun 24 10:27:15 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 24 10:27:15 fir-md1-s1 kernel: Lustre: 97654:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:129s); client may timeout. req@ffff8f1d5bba9200 x1633783014311664/t0(0) o101->19313a8c-b11b-17b1-39e1-85aeb6c20cba@10.8.15.9@o2ib6:6/0 lens 1768/0 e 1 to 0 dl 1561397106 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jun 24 10:27:16 fir-md1-s1 kernel: LustreError: 50448:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f24ee52e000 ns: mdt-fir-MDT0000_UUID lock: ffff8f23ebb60fc0/0x5d9ee622c3bc50dc lrc: 3/0,0 mode: PW/PW res: [0x200029bbb:0xd:0x0].0x0 bits 0x40/0x0 rrc: 250 type: IBT flags: 0x50200400000020 nid: 10.8.10.20@o2ib6 remote: 0xc48f9d87344ea8c4 expref: 17 pid: 50448 timeout: 0 lvb_type: 0 Jun 24 10:27:16 fir-md1-s1 kernel: LustreError: 24577:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.16.5@o2ib6: deadline 20:1s ago req@ffff8f17e2647500 x1634924007495888/t0(0) o101->1fb1c1bc-a5c2-7639-1248-10341b490c82@10.8.16.5@o2ib6:14/0 lens 1768/0 e 0 to 0 dl 1561397234 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jun 24 10:27:16 fir-md1-s1 kernel: LustreError: 24577:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 48 previous similar messages Jun 24 10:27:16 fir-md1-s1 kernel: Lustre: 97654:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2747 previous similar messages Jun 24 10:27:19 fir-md1-s1 kernel: LustreError: 97642:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f24ee52e000 ns: mdt-fir-MDT0000_UUID lock: ffff8f24f0ed33c0/0x5d9ee622c3bc65b2 lrc: 3/0,0 mode: PW/PW res: [0x200029bbb:0xd:0x0].0x0 bits 0x40/0x0 rrc: 174 type: IBT flags: 0x50200400000020 nid: 10.8.10.20@o2ib6 remote: 0xc48f9d87344ea949 expref: 10 pid: 97642 timeout: 0 lvb_type: 0 Jun 24 10:27:19 fir-md1-s1 kernel: LustreError: 97642:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages Jun 24 10:27:34 fir-md1-s1 kernel: Lustre: 20721:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (167:1s); client may timeout. req@ffff8f18887a8f00 x1631646255656320/t0(0) o101->f03aa5e8-f764-2262-c217-2e99830bfe5f@10.8.22.34@o2ib6:6/0 lens 480/536 e 1 to 0 dl 1561397253 ref 1 fl Complete:/0/0 rc 0/0 Jun 24 10:27:34 fir-md1-s1 kernel: LustreError: 20724:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f24ee52e000 ns: mdt-fir-MDT0000_UUID lock: ffff8f16b4cb2d00/0x5d9ee622c3bc6bcb lrc: 3/0,0 mode: PW/PW res: [0x200029bbb:0xd:0x0].0x0 bits 0x40/0x0 rrc: 147 type: IBT flags: 0x50200400000020 nid: 10.8.10.20@o2ib6 remote: 0xc48f9d87344ea981 expref: 8 pid: 20724 timeout: 0 lvb_type: 0 Jun 24 10:27:34 fir-md1-s1 kernel: Lustre: 20721:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 16 previous similar messages Jun 24 12:34:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ffa27290-6cf4-9b77-ab2a-7df1aa693fad (at 10.8.21.21@o2ib6) Jun 24 12:34:54 fir-md1-s1 kernel: Lustre: Skipped 147 previous similar messages Jun 24 12:35:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client cfd4e192-da61-c95f-6005-fc026e176bd8 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2520744400, cur 1561404903 expire 1561404753 last 1561404676 Jun 24 13:05:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ffa27290-6cf4-9b77-ab2a-7df1aa693fad (at 10.8.21.21@o2ib6) Jun 24 13:05:16 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 13:05:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 460d4bd1-5320-0f4d-604d-3fee0115b165 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1a0fef1000, cur 1561406728 expire 1561406578 last 1561406501 Jun 24 13:05:28 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 14:22:51 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561411364/real 1561411364] req@ffff8f07d12de000 x1636713186879872/t0(0) o104->fir-MDT0002@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561411371 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 24 14:22:51 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Jun 24 14:22:59 fir-md1-s1 kernel: Lustre: 23602:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1ee9503900 x1636449140881520/t0(0) o36->59f098aa-fb21-8ed8-84bd-d0ce06cad654@10.9.102.46@o2ib4:4/0 lens 520/448 e 1 to 0 dl 1561411384 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 14:22:59 fir-md1-s1 kernel: Lustre: 23602:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 413 previous similar messages Jun 24 14:23:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 59f098aa-fb21-8ed8-84bd-d0ce06cad654 (at 10.9.102.46@o2ib4) reconnecting Jun 24 14:23:05 fir-md1-s1 kernel: Lustre: Skipped 269 previous similar messages Jun 24 14:23:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 460b4624-f225-0fc6-9d6f-aee495221c30 (at 10.9.102.46@o2ib4) Jun 24 14:23:05 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 14:23:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 826fbeb7-54e9-5127-860e-c32891bc78a7 (at 10.9.107.9@o2ib4) Jun 24 14:23:06 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 14:23:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to d90e9165-328c-67de-acd1-290e1860ac02 (at 10.8.16.7@o2ib6) Jun 24 14:23:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 8ab6533c-237c-52d9-a0d0-b7b0b3591cd2 (at 10.9.108.34@o2ib4) Jun 24 14:23:10 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 14:23:12 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561411385/real 1561411385] req@ffff8f07d12de000 x1636713186879872/t0(0) o104->fir-MDT0002@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561411392 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 24 14:23:12 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 24 14:23:16 fir-md1-s1 kernel: Lustre: 20721:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f215f817200 x1631561721028144/t0(0) o101->b4e75cd9-74c7-0ec8-2651-b87e466f256d@10.9.105.70@o2ib4:21/0 lens 576/3264 e 1 to 0 dl 1561411401 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 14:23:16 fir-md1-s1 kernel: Lustre: 20721:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 13 previous similar messages Jun 24 14:23:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to dce245ee-1721-1fa3-f0f5-8ef6b7994bca (at 10.9.105.27@o2ib4) Jun 24 14:23:16 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jun 24 14:23:19 fir-md1-s1 kernel: LustreError: 10506:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.9@o2ib6) failed to reply to blocking AST (req@ffff8f07d12de000 x1636713186879872 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f2183e38d80/0x5d9ee62316112cd2 lrc: 4/0,0 mode: PR/PR res: [0x2c0024163:0x19838:0x0].0x0 bits 0x13/0x0 rrc: 342 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x808140e2f7ea8097 expref: 650 pid: 22007 timeout: 526481 lvb_type: 0 Jun 24 14:23:19 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.9.9@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jun 24 14:23:19 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jun 24 14:23:19 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2183e38d80/0x5d9ee62316112cd2 lrc: 3/0,0 mode: PR/PR res: [0x2c0024163:0x19838:0x0].0x0 bits 0x13/0x0 rrc: 342 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x808140e2f7ea8097 expref: 651 pid: 22007 timeout: 0 lvb_type: 0 Jun 24 14:23:19 fir-md1-s1 kernel: Lustre: 23644:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:1s); client may timeout. req@ffff8f0c52e23f00 x1631537617276800/t0(0) o101->7384665e-bddc-c186-a2f8-10bf76931a32@10.9.106.44@o2ib4:18/0 lens 576/536 e 1 to 0 dl 1561411398 ref 1 fl Complete:/0/0 rc 0/0 Jun 24 14:23:19 fir-md1-s1 kernel: Lustre: 23644:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages Jun 24 14:24:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 24 14:24:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 14:24:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 804bb2d0-a656-6c01-b0db-5b53058fb0f9 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24efef9400, cur 1561411469 expire 1561411319 last 1561411242 Jun 24 14:24:29 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 14:24:43 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f253b992000, cur 1561411483 expire 1561411333 last 1561411256 Jun 24 15:00:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.0.64@o2ib4, removing former export from same NID Jun 24 15:00:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 24 15:00:06 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jun 24 15:00:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 24 15:00:06 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 15:00:06 fir-md1-s1 kernel: Lustre: Skipped 233 previous similar messages Jun 24 15:00:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 24 15:00:11 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 24 15:00:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 24 15:00:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 24 15:00:19 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:00:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 24 15:00:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 24 15:00:36 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:01:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.0.64@o2ib4, removing former export from same NID Jun 24 15:01:13 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:01:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 24 15:01:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 24 15:01:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 24 15:01:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 24 15:01:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:01:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:02:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 24 15:02:57 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:02:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 24 15:02:57 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jun 24 15:15:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) reconnecting Jun 24 15:15:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) Jun 24 15:15:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID Jun 24 15:15:05 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 15:15:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 24 15:15:24 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 24 15:15:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID Jun 24 15:15:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) reconnecting Jun 24 15:15:49 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:15:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) Jun 24 15:15:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) Jun 24 15:15:49 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 15:15:49 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 24 15:17:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jun 24 15:17:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) Jun 24 15:17:40 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:23:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jun 24 15:23:15 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:23:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) Jun 24 15:23:15 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:23:15 fir-md1-s1 kernel: LustreError: 25997:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f2521cd0c50 x1634306077191856/t0(0) o4->a6b91a43-6f67-a7e7-0e97-a87e8033e0cf@10.8.9.10@o2ib6:9/0 lens 488/448 e 0 to 0 dl 1561415019 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:23:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO write error with a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6), client will retry: rc = -110 Jun 24 15:23:15 fir-md1-s1 kernel: LustreError: 25997:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message Jun 24 15:23:44 fir-md1-s1 kernel: Lustre: 23455:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f214b22aa00 x1636441686336256/t0(0) o101->9eed212b-34d9-6e26-f1ac-cdc452decf97@10.8.29.3@o2ib6:19/0 lens 376/1600 e 1 to 0 dl 1561415029 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:23:44 fir-md1-s1 kernel: Lustre: 23455:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jun 24 15:42:31 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:42:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 13458280-a046-3a7f-2bec-0301aba013a1 (at 10.8.28.12@o2ib6) reconnecting Jun 24 15:42:38 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:42:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to d0d1dcda-abd5-29f1-1250-5971b6db7d8a (at 10.8.28.12@o2ib6) Jun 24 15:42:38 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:46:17 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:46:17 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Jun 24 15:46:22 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:46:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 7b7e9b9d-7d80-a5c4-07fd-dd92cbcbe2f0 (at 10.8.29.6@o2ib6) reconnecting Jun 24 15:46:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:46:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 0af4f40a-317e-88ce-7d9c-c4839b78e5a4 (at 10.8.29.6@o2ib6) Jun 24 15:46:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:46:24 fir-md1-s1 kernel: LustreError: 21543:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8f1ef3ac9450 x1636443218314096/t0(0) o3->7b7e9b9d-7d80-a5c4-07fd-dd92cbcbe2f0@10.8.29.6@o2ib6:23/0 lens 488/440 e 0 to 0 dl 1561416413 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:46:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 7b7e9b9d-7d80-a5c4-07fd-dd92cbcbe2f0 (at 10.8.29.6@o2ib6), client will retry: rc -110 Jun 24 15:46:27 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:46:27 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages Jun 24 15:46:32 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:46:32 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages Jun 24 15:46:42 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:46:42 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 12 previous similar messages Jun 24 15:46:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 9081d826-2f83-5b46-ff73-7e6473184838 (at 10.8.17.25@o2ib6) reconnecting Jun 24 15:46:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 420c129b-df9e-b1c5-eae5-667fed64bb9d (at 10.8.15.3@o2ib6) Jun 24 15:46:57 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jun 24 15:46:58 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:46:58 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 24 previous similar messages Jun 24 15:46:59 fir-md1-s1 kernel: LustreError: 46578:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1974bae450 x1631566216131696/t0(0) o4->be42b497-ab1b-8d58-3101-014aad577cfc@10.8.27.35@o2ib6:26/0 lens 488/448 e 0 to 0 dl 1561416446 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:47:01 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1cd303a000 Jun 24 15:47:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with be42b497-ab1b-8d58-3101-014aad577cfc (at 10.8.27.35@o2ib6), client will retry: rc = -110 Jun 24 15:47:01 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:47:03 fir-md1-s1 kernel: LustreError: 27583:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f24e889dc50 x1631557242040512/t0(0) o4->84fd8c4b-6545-cd41-282d-ef5f651cba30@10.8.17.11@o2ib6:29/0 lens 488/448 e 0 to 0 dl 1561416449 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:47:03 fir-md1-s1 kernel: LustreError: 27583:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message Jun 24 15:47:04 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f13b622f400 Jun 24 15:47:04 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f22bf91e000 Jun 24 15:47:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6), client will retry: rc = -110 Jun 24 15:47:13 fir-md1-s1 kernel: LustreError: 22730:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1f9946ec50 x1631538709023600/t0(0) o4->ca15d879-1cb2-8780-e5e2-20230d9e27cf@10.8.28.3@o2ib6:10/0 lens 488/448 e 0 to 0 dl 1561416460 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:47:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 8e6b7782-0f04-da33-0138-eab1c9e41ffb (at 10.8.18.25@o2ib6) reconnecting Jun 24 15:47:16 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jun 24 15:47:16 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1b0b10c800 Jun 24 15:47:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with ca15d879-1cb2-8780-e5e2-20230d9e27cf (at 10.8.28.3@o2ib6), client will retry: rc = -110 Jun 24 15:47:16 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:47:17 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e3f2bf600 Jun 24 15:47:25 fir-md1-s1 kernel: Lustre: 21433:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561416438/real 0] req@ffff8f18dbffad00 x1636713474198768/t0(0) o104->fir-MDT0002@10.8.8.17@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561416445 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 24 15:47:25 fir-md1-s1 kernel: Lustre: 21433:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 24 15:47:31 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:47:31 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 41 previous similar messages Jun 24 15:47:33 fir-md1-s1 kernel: Lustre: 97672:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561416446/real 0] req@ffff8f1a75e0bf00 x1636713474219552/t0(0) o104->fir-MDT0000@10.8.29.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561416453 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 24 15:47:33 fir-md1-s1 kernel: Lustre: 97672:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jun 24 15:47:43 fir-md1-s1 kernel: LustreError: 46578:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8f2521cd4450 x1636443218401408/t0(0) o3->7b7e9b9d-7d80-a5c4-07fd-dd92cbcbe2f0@10.8.29.6@o2ib6:12/0 lens 488/440 e 0 to 0 dl 1561416492 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:47:43 fir-md1-s1 kernel: LustreError: 46578:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message Jun 24 15:47:47 fir-md1-s1 kernel: Lustre: 21460:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561416460/real 0] req@ffff8f251dbab300 x1636713474283296/t0(0) o104->fir-MDT0000@10.8.29.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561416467 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 24 15:47:48 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1696a6a600 Jun 24 15:47:48 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f192fb0f400 Jun 24 15:47:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 13458280-a046-3a7f-2bec-0301aba013a1 (at 10.8.28.12@o2ib6), client will retry: rc = -110 Jun 24 15:47:48 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:47:52 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d7b9b0800 Jun 24 15:47:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 7b7e9b9d-7d80-a5c4-07fd-dd92cbcbe2f0 (at 10.8.29.6@o2ib6), client will retry: rc -110 Jun 24 15:47:54 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f22443e7c00 Jun 24 15:47:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6), client will retry: rc = -110 Jun 24 15:47:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:48:00 fir-md1-s1 kernel: Lustre: 23660:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f19eaf2f500 x1636449131623072/t0(0) o36->9d52b61d-61c3-c5c4-3713-7cb415666394@10.9.102.34@o2ib4:5/0 lens 520/448 e 1 to 0 dl 1561416485 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:48:02 fir-md1-s1 kernel: Lustre: 23743:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f29b6764e00 x1631563634202912/t0(0) o101->3ef17f0c-d35b-8428-c1da-c84a40a8bdbc@10.9.101.71@o2ib4:7/0 lens 576/3264 e 1 to 0 dl 1561416487 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:48:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 327c2a50-dba2-1c9c-0f3d-801872275c5c (at 10.8.18.26@o2ib6) Jun 24 15:48:05 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jun 24 15:48:10 fir-md1-s1 kernel: LustreError: 25630:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f2505939050 x1631301640417632/t0(0) o4->6e0b1c17-2142-9190-acc8-624208298012@10.8.8.17@o2ib6:1/0 lens 488/448 e 0 to 0 dl 1561416511 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:48:10 fir-md1-s1 kernel: LustreError: 25630:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 3 previous similar messages Jun 24 15:48:13 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f44e6a79800 Jun 24 15:48:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 6e0b1c17-2142-9190-acc8-624208298012 (at 10.8.8.17@o2ib6), client will retry: rc = -110 Jun 24 15:48:18 fir-md1-s1 kernel: Lustre: 50446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561416491/real 0] req@ffff8f168f37f200 x1636713474396208/t0(0) o104->fir-MDT0002@10.8.7.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561416498 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 24 15:48:18 fir-md1-s1 kernel: Lustre: 50446:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 59 previous similar messages Jun 24 15:48:19 fir-md1-s1 kernel: Lustre: 21433:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f16770ab300 x1631538709029264/t0(0) o101->ca15d879-1cb2-8780-e5e2-20230d9e27cf@10.8.28.3@o2ib6:24/0 lens 576/3264 e 0 to 0 dl 1561416504 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:48:19 fir-md1-s1 kernel: Lustre: 21433:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jun 24 15:48:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b37c54be-7fed-724b-d760-c5bd71b2a4e0 (at 10.8.29.5@o2ib6) reconnecting Jun 24 15:48:24 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jun 24 15:48:30 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1ec0abd600 Jun 24 15:48:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO write error with b37c54be-7fed-724b-d760-c5bd71b2a4e0 (at 10.8.29.5@o2ib6), client will retry: rc = -110 Jun 24 15:48:32 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f222778b400 Jun 24 15:48:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO read error with b37c54be-7fed-724b-d760-c5bd71b2a4e0 (at 10.8.29.5@o2ib6), client will retry: rc -110 Jun 24 15:48:36 fir-md1-s1 kernel: LustreError: 22156:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1f77339450 x1636570040419104/t0(0) o4->a6d577d8-fd68-2a67-a952-7c8d9e354cb8@10.8.8.24@o2ib6:2/0 lens 488/448 e 0 to 0 dl 1561416542 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:48:36 fir-md1-s1 kernel: LustreError: 22156:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 2 previous similar messages Jun 24 15:48:37 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:48:37 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 52 previous similar messages Jun 24 15:48:44 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f17e7b06200 Jun 24 15:48:47 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1595ef9200 Jun 24 15:48:55 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1a75e08000 Jun 24 15:48:55 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f160e218400 Jun 24 15:48:55 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f17ab7d0000 Jun 24 15:48:57 fir-md1-s1 kernel: Lustre: 21368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561416530/real 0] req@ffff8f10419b1800 x1636713474551312/t0(0) o106->fir-MDT0000@10.8.27.24@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561416537 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 24 15:48:57 fir-md1-s1 kernel: Lustre: 21368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 24 15:49:07 fir-md1-s1 kernel: Lustre: 97670:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f163b712400 x1635340613822832/t0(0) o101->c1c54f8a-db68-72ea-1f4f-3dc905e7ab7d@10.8.1.16@o2ib6:12/0 lens 480/568 e 0 to 0 dl 1561416552 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:49:07 fir-md1-s1 kernel: Lustre: 97670:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jun 24 15:49:08 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1635fbe200 Jun 24 15:49:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO write error with 00a6bf4a-1a11-675b-07eb-2392e93c70c7 (at 10.8.29.8@o2ib6), client will retry: rc = -110 Jun 24 15:49:08 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jun 24 15:49:15 fir-md1-s1 kernel: LustreError: 22648:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1a8aff3050 x1636418132959360/t0(0) o4->304180e1-aa68-a4a4-ed4c-9536f53351a5@10.8.1.21@o2ib6:9/0 lens 488/448 e 0 to 0 dl 1561416579 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:49:15 fir-md1-s1 kernel: LustreError: 22648:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 7 previous similar messages Jun 24 15:49:15 fir-md1-s1 kernel: Lustre: 23556:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f06ce310c00 x1636996283584528/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:20/0 lens 480/568 e 0 to 0 dl 1561416560 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:49:21 fir-md1-s1 kernel: Lustre: 21368:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:1s); client may timeout. req@ffff8f06ce310c00 x1636996283584528/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:20/0 lens 480/536 e 0 to 0 dl 1561416560 ref 1 fl Complete:/0/0 rc 301/301 Jun 24 15:49:27 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f243d644200 Jun 24 15:49:27 fir-md1-s1 kernel: Lustre: 97658:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1667215100 x1631309731138176/t0(0) o101->2defae61-8bf0-dee6-7d48-53b83a69e973@10.8.17.24@o2ib6:2/0 lens 1808/3288 e 0 to 0 dl 1561416572 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:49:27 fir-md1-s1 kernel: Lustre: 97658:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jun 24 15:49:27 fir-md1-s1 kernel: Lustre: 24578:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:3s); client may timeout. req@ffff8f160ae8a400 x1631557242055184/t348692003537(0) o101->84fd8c4b-6545-cd41-282d-ef5f651cba30@10.8.17.11@o2ib6:24/0 lens 1776/1192 e 0 to 0 dl 1561416564 ref 1 fl Complete:/0/0 rc 0/0 Jun 24 15:49:27 fir-md1-s1 kernel: Lustre: 24578:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jun 24 15:49:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.18.1@o2ib6, removing former export from same NID Jun 24 15:49:32 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1adebcda00 Jun 24 15:49:32 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f162bb70600 Jun 24 15:49:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 62873e5a-5401-394e-2139-5fd47462d1df (at 10.8.29.2@o2ib6), client will retry: rc -110 Jun 24 15:49:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.20.11@o2ib6, removing former export from same NID Jun 24 15:49:34 fir-md1-s1 kernel: Lustre: Skipped 112 previous similar messages Jun 24 15:49:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.7.12@o2ib6, removing former export from same NID Jun 24 15:49:42 fir-md1-s1 kernel: Lustre: Skipped 187 previous similar messages Jun 24 15:49:42 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f167abf0c00 Jun 24 15:49:42 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1629310600 Jun 24 15:49:43 fir-md1-s1 kernel: Lustre: 22283:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f217a561e00 x1631575960598752/t0(0) o101->4dc6ad45-c67c-15d0-5638-611b0defe5f9@10.8.16.2@o2ib6:18/0 lens 376/1600 e 0 to 0 dl 1561416588 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:49:43 fir-md1-s1 kernel: Lustre: 22283:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 18 previous similar messages Jun 24 15:49:48 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1fd244c200 Jun 24 15:49:50 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1cc715be00 Jun 24 15:49:52 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f3091efac00 Jun 24 15:49:53 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f25311d5e00 Jun 24 15:49:54 fir-md1-s1 kernel: Lustre: 97670:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:15s); client may timeout. req@ffff8f19f1ad8300 x1636669927723920/t348691996062(0) o36->cea6adbc-46ce-842f-a429-3350fc5db284@10.8.18.26@o2ib6:9/0 lens 488/424 e 0 to 0 dl 1561416579 ref 1 fl Complete:/0/0 rc 0/0 Jun 24 15:49:56 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.27.22@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1cfcaf8900/0x5d9ee6233f217b96 lrc: 3/0,0 mode: PR/PR res: [0x2c0014fbb:0x115fc:0x0].0x0 bits 0x13/0x0 rrc: 97 type: IBT flags: 0x60200400000020 nid: 10.8.27.22@o2ib6 remote: 0x4deb3a7a8dd7d1fe expref: 345 pid: 97645 timeout: 531656 lvb_type: 0 Jun 24 15:49:57 fir-md1-s1 kernel: Lustre: 21456:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:24s); client may timeout. req@ffff8f1667217500 x1635086170030736/t348692019075(0) o101->bc83c7c5-08aa-b1e5-1dd5-b1a51ba5cb4a@10.8.1.15@o2ib6:2/0 lens 1776/1192 e 0 to 0 dl 1561416572 ref 1 fl Complete:/0/0 rc 0/0 Jun 24 15:49:57 fir-md1-s1 kernel: Lustre: 21456:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages Jun 24 15:49:58 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1970efe600 Jun 24 15:49:58 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20532ca400 Jun 24 15:49:59 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.9@o2ib6, removing former export from same NID Jun 24 15:49:59 fir-md1-s1 kernel: Lustre: Skipped 323 previous similar messages Jun 24 15:49:59 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f29b4673200 Jun 24 15:50:02 fir-md1-s1 kernel: Lustre: 97645:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561416592/real 0] req@ffff8f23597db900 x1636713474808752/t0(0) o104->fir-MDT0002@10.8.17.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561416602 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 24 15:50:02 fir-md1-s1 kernel: Lustre: 97645:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 31 previous similar messages Jun 24 15:50:02 fir-md1-s1 kernel: Lustre: 97671:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:18s); client may timeout. req@ffff8f1624b33000 x1631595884900736/t0(0) o101->40db60e6-2b5f-e52d-2610-43b84e2f829d@10.8.29.1@o2ib6:14/0 lens 376/944 e 0 to 0 dl 1561416584 ref 1 fl Complete:/0/0 rc 0/0 Jun 24 15:50:02 fir-md1-s1 kernel: Lustre: 97671:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages Jun 24 15:50:04 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e4de16000 Jun 24 15:50:05 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 51s: evicting client at 10.8.8.24@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f21d5d660c0/0x5d9ee6233b58597c lrc: 3/0,0 mode: PR/PR res: [0x2c002bf5a:0x5c34:0x0].0x0 bits 0x5b/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 10.8.8.24@o2ib6 remote: 0xc0455945f6b89b52 expref: 11824 pid: 20730 timeout: 531665 lvb_type: 0 Jun 24 15:50:05 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 24 15:50:10 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 52s: evicting client at 10.8.16.2@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1f5301bf00/0x5d9ee6233e9230c5 lrc: 3/0,0 mode: CR/CR res: [0x2c002be48:0x104df:0x0].0x0 bits 0x9/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.16.2@o2ib6 remote: 0x24cf00c2f87a7f94 expref: 1910 pid: 26256 timeout: 531670 lvb_type: 0 Jun 24 15:50:10 fir-md1-s1 kernel: LustreError: 24579:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f2501a6b400 ns: mdt-fir-MDT0002_UUID lock: ffff8f2ceea72f40/0x5d9ee6233f3f88e3 lrc: 1/0,0 mode: EX/EX res: [0x2c002be48:0x104df:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT flags: 0x54801000000000 nid: 10.8.16.2@o2ib6 remote: 0x24cf00c2f87a8004 expref: 1296 pid: 24579 timeout: 0 lvb_type: 3 Jun 24 15:50:10 fir-md1-s1 kernel: LustreError: 24579:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 3 previous similar messages Jun 24 15:50:11 fir-md1-s1 kernel: Lustre: 24579:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:22s); client may timeout. req@ffff8f217a561e00 x1631575960598752/t348692025511(0) o101->4dc6ad45-c67c-15d0-5638-611b0defe5f9@10.8.16.2@o2ib6:18/0 lens 376/1568 e 0 to 0 dl 1561416588 ref 1 fl Complete:/0/0 rc -107/-107 Jun 24 15:50:11 fir-md1-s1 kernel: Lustre: 24579:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jun 24 15:50:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to f590aa0d-878d-f7af-2791-1d94ccac0e1f (at 10.8.18.1@o2ib6) Jun 24 15:50:13 fir-md1-s1 kernel: Lustre: Skipped 769 previous similar messages Jun 24 15:50:13 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1653697c00 Jun 24 15:50:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with cea6adbc-46ce-842f-a429-3350fc5db284 (at 10.8.18.26@o2ib6), client will retry: rc = -110 Jun 24 15:50:13 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jun 24 15:50:15 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 54s: evicting client at 10.8.29.5@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f19a86357c0/0x5d9ee6233ea985f5 lrc: 3/0,0 mode: PR/PR res: [0x20002993d:0x274:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x60000400000020 nid: 10.8.29.5@o2ib6 remote: 0xc606c8a810cda247 expref: 104 pid: 22007 timeout: 531675 lvb_type: 0 Jun 24 15:50:15 fir-md1-s1 kernel: Lustre: 97660:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f163caff200 x1631695367410864/t0(0) o101->e0767d77-866c-9038-3794-0af657e399d1@10.8.8.22@o2ib6:20/0 lens 1936/3288 e 0 to 0 dl 1561416620 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:50:15 fir-md1-s1 kernel: Lustre: 97660:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 50 previous similar messages Jun 24 15:50:16 fir-md1-s1 kernel: LustreError: 23455:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f34ed706400 ns: mdt-fir-MDT0000_UUID lock: ffff8f232b219200/0x5d9ee6233f441c8a lrc: 3/0,0 mode: PW/PW res: [0x20002993d:0x274:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x50200000000000 nid: 10.8.29.5@o2ib6 remote: 0xc606c8a810cda2e8 expref: 79 pid: 23455 timeout: 0 lvb_type: 0 Jun 24 15:50:19 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f163ff40600 Jun 24 15:50:26 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.17.11@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f21d7d81d40/0x5d9ee6233f32b986 lrc: 3/0,0 mode: PR/PR res: [0x2c0014fbb:0x115fc:0x0].0x0 bits 0x13/0x0 rrc: 82 type: IBT flags: 0x60200400000020 nid: 10.8.17.11@o2ib6 remote: 0x23a0b048f5b281f7 expref: 749 pid: 97638 timeout: 531686 lvb_type: 0 Jun 24 15:50:28 fir-md1-s1 kernel: LustreError: 46534:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1a8aff3c50 x1631543243076544/t0(0) o4->20ffa3e6-2ce8-ff35-0cee-96ba2468fd67@10.8.17.13@o2ib6:12/0 lens 488/448 e 0 to 0 dl 1561416642 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:50:28 fir-md1-s1 kernel: LustreError: 46534:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 14 previous similar messages Jun 24 15:50:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.27.1@o2ib6, removing former export from same NID Jun 24 15:50:31 fir-md1-s1 kernel: Lustre: Skipped 337 previous similar messages Jun 24 15:50:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 044042bf-dd57-7ee7-fd56-cb18003c928b (at 10.8.7.32@o2ib6) reconnecting Jun 24 15:50:34 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jun 24 15:50:36 fir-md1-s1 kernel: Lustre: 97667:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:23s); client may timeout. req@ffff8f1bd273e600 x1631562893236800/t0(0) o101->69e867f7-2c34-9281-0411-6ff880d43ef5@10.8.28.11@o2ib6:13/0 lens 384/1040 e 0 to 0 dl 1561416613 ref 1 fl Complete:/0/0 rc 0/0 Jun 24 15:50:36 fir-md1-s1 kernel: Lustre: 97667:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 7 previous similar messages Jun 24 15:50:37 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f237fb96400 Jun 24 15:50:40 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f167abf1400 Jun 24 15:50:41 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20968b2e00 Jun 24 15:50:44 fir-md1-s1 kernel: LustreError: 97669:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561416554, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1ce7674380/0x5d9ee6233f3ad679 lrc: 3/0,1 mode: --/CW res: [0x2c0014fbb:0x115fc:0x0].0x0 bits 0x2/0x0 rrc: 75 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 97669 timeout: 0 lvb_type: 0 Jun 24 15:50:45 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 24 15:50:45 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 90 previous similar messages Jun 24 15:50:47 fir-md1-s1 kernel: LustreError: 22289:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561416557, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1706f3da00/0x5d9ee6233f3e7aa6 lrc: 3/1,0 mode: --/PR res: [0x2c0014fbb:0x115fc:0x0].0x0 bits 0x13/0x0 rrc: 75 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 22289 timeout: 0 lvb_type: 0 Jun 24 15:50:47 fir-md1-s1 kernel: LustreError: 22289:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages Jun 24 15:50:48 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 56s: evicting client at 10.8.27.3@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f3fd871ad00/0x5d9ee6233efe9fae lrc: 3/0,0 mode: PR/PR res: [0x2c002bea6:0x1e36b:0x0].0x0 bits 0x13/0x0 rrc: 80 type: IBT flags: 0x60200400000020 nid: 10.8.27.3@o2ib6 remote: 0xf651ae946746c380 expref: 129 pid: 20722 timeout: 531708 lvb_type: 0 Jun 24 15:50:49 fir-md1-s1 kernel: LustreError: 50445:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561416559, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f21aa0086c0/0x5d9ee6233f412f0c lrc: 3/1,0 mode: --/PR res: [0x2c0014fbb:0x115fc:0x0].0x0 bits 0x13/0x0 rrc: 73 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 50445 timeout: 0 lvb_type: 0 Jun 24 15:50:50 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1626724600 Jun 24 15:50:56 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1f3201a000 Jun 24 15:51:03 fir-md1-s1 kernel: LustreError: 25082:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.7.32@o2ib6 arrived at 1561416663 with bad export cookie 6746082289100437273 Jun 24 15:51:06 fir-md1-s1 kernel: LustreError: 21712:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 30+0s req@ffff8f1a8aff1450 x1637258515994832/t0(0) o3->b09d4c25-b109-b30c-132e-6a644105be34@10.8.9.9@o2ib6:6/0 lens 488/440 e 0 to 0 dl 1561416666 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:51:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO read error with b09d4c25-b109-b30c-132e-6a644105be34 (at 10.8.9.9@o2ib6), client will retry: rc -110 Jun 24 15:51:06 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 24 15:51:07 fir-md1-s1 kernel: LustreError: 42895:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 30+0s req@ffff8f21343b8450 x1634352865824384/t0(0) o4->eb079895-c48f-19eb-1198-2b2f152dbaf1@10.8.26.34@o2ib6:7/0 lens 488/448 e 0 to 0 dl 1561416667 ref 1 fl Interpret:/2/0 rc 0/0 Jun 24 15:51:08 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1a6880a400 Jun 24 15:51:08 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f161ff6dc00 Jun 24 15:51:08 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d7b9b7e00 Jun 24 15:51:08 fir-md1-s1 kernel: LustreError: 21449:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 30+0s req@ffff8f1dcf7f8c50 x1631567245092016/t0(0) o4->c85b79ba-f35a-df4c-7ce6-3db4837c1dc9@10.8.18.1@o2ib6:8/0 lens 488/448 e 0 to 0 dl 1561416668 ref 1 fl Interpret:/2/0 rc 0/0 Jun 24 15:51:08 fir-md1-s1 kernel: LustreError: 21449:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 2 previous similar messages Jun 24 15:51:09 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1696a6fc00 Jun 24 15:51:09 fir-md1-s1 kernel: Lustre: 21449:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:1s); client may timeout. req@ffff8f1dcf7f8c50 x1631567245092016/t0(0) o4->c85b79ba-f35a-df4c-7ce6-3db4837c1dc9@10.8.18.1@o2ib6:8/0 lens 488/448 e 0 to 0 dl 1561416668 ref 1 fl Complete:/2/ffffffff rc -110/-1 Jun 24 15:51:09 fir-md1-s1 kernel: Lustre: 21449:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 11 previous similar messages Jun 24 15:51:10 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f16a2e9d800 Jun 24 15:51:10 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f21437a6e00 Jun 24 15:51:11 fir-md1-s1 kernel: LustreError: 22648:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 30+0s req@ffff8f1f7733ac50 x1636569714132368/t0(0) o4->5d60b790-0b15-ff01-65b5-d8a0250b0e53@10.8.1.29@o2ib6:11/0 lens 488/448 e 0 to 0 dl 1561416671 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:51:11 fir-md1-s1 kernel: LustreError: 22648:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message Jun 24 15:51:14 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2348eaa400 Jun 24 15:51:15 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1891e22000 Jun 24 15:51:15 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1f746adc00 Jun 24 15:51:16 fir-md1-s1 kernel: LustreError: 20461:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.29.6@o2ib6) failed to reply to blocking AST (req@ffff8f161ca61b00 x1636713475012736 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f22fdbb0000/0x5d9ee6233f474ec6 lrc: 4/0,0 mode: EX/EX res: [0x2c002bf84:0x9313:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT flags: 0x60000400000020 nid: 10.8.29.6@o2ib6 remote: 0xcb7f8716e1872de0 expref: 14900 pid: 22004 timeout: 531732 lvb_type: 3 Jun 24 15:51:16 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.29.6@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jun 24 15:51:16 fir-md1-s1 kernel: LustreError: 27583:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 30+0s req@ffff8f1ee25a3450 x1631683569568144/t0(0) o4->a82097ea-0a83-cc99-985b-882074216844@10.8.12.13@o2ib6:16/0 lens 504/448 e 0 to 0 dl 1561416676 ref 1 fl Interpret:/2/0 rc 0/0 Jun 24 15:51:16 fir-md1-s1 kernel: LustreError: 27583:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 4 previous similar messages Jun 24 15:51:16 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1df1688400 Jun 24 15:51:17 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20cd79f400 Jun 24 15:51:19 fir-md1-s1 kernel: Lustre: 25998:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1974ba8050 x1634927199608064/t0(0) o4->8e6b7782-0f04-da33-0138-eab1c9e41ffb@10.8.18.25@o2ib6:24/0 lens 488/448 e 0 to 0 dl 1561416684 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:51:19 fir-md1-s1 kernel: Lustre: 25998:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 120 previous similar messages Jun 24 15:51:20 fir-md1-s1 kernel: LustreError: 24584:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561416590, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f216ce20000/0x5d9ee6233f6e6dd5 lrc: 3/0,1 mode: --/CW res: [0x2c0014fbb:0x115fc:0x0].0x0 bits 0x2/0x0 rrc: 69 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 24584 timeout: 0 lvb_type: 0 Jun 24 15:51:20 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1ece1a0200 Jun 24 15:51:23 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.8.17@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1706f3da00/0x5d9ee6233f3e7aa6 lrc: 3/0,0 mode: PR/PR res: [0x2c0014fbb:0x115fc:0x0].0x0 bits 0x13/0x0 rrc: 71 type: IBT flags: 0x60200400000020 nid: 10.8.8.17@o2ib6 remote: 0x68316722491f52a3 expref: 2391 pid: 22289 timeout: 531743 lvb_type: 0 Jun 24 15:51:23 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 24 15:51:24 fir-md1-s1 kernel: LustreError: 83752:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f3438f70c00 x1636713475180944/t0(0) o105->fir-MDT0002@10.8.28.11@o2ib6:15/16 lens 304/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 24 15:51:24 fir-md1-s1 kernel: LustreError: 21434:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f25350ae800 ns: mdt-fir-MDT0002_UUID lock: ffff8f3227283f00/0x5d9ee6233f70e4d1 lrc: 3/0,0 mode: PR/PR res: [0x2c0014fbb:0x115fc:0x0].0x0 bits 0x13/0x0 rrc: 55 type: IBT flags: 0x50200000000000 nid: 10.8.28.3@o2ib6 remote: 0x8a5f985bbadec0dc expref: 7 pid: 21434 timeout: 0 lvb_type: 0 Jun 24 15:51:24 fir-md1-s1 kernel: LustreError: 21497:0:(ldlm_lib.c:3252:target_bulk_io()) @@@ Eviction on bulk WRITE req@ffff8f180f6a2c50 x1631538709055328/t0(0) o4->ca15d879-1cb2-8780-e5e2-20230d9e27cf@10.8.28.3@o2ib6:16/0 lens 488/448 e 0 to 0 dl 1561416706 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:51:25 fir-md1-s1 kernel: LustreError: 46590:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 30+0s req@ffff8f1974ba8050 x1634927199608064/t0(0) o4->8e6b7782-0f04-da33-0138-eab1c9e41ffb@10.8.18.25@o2ib6:24/0 lens 488/448 e 0 to 0 dl 1561416684 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:51:25 fir-md1-s1 kernel: LustreError: 46590:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 9 previous similar messages Jun 24 15:51:25 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f209d4b1c00 Jun 24 15:51:25 fir-md1-s1 kernel: LustreError: 22136:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.28.11@o2ib6 arrived at 1561416685 with bad export cookie 6746082289092222801 Jun 24 15:51:26 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1666b9ce00 Jun 24 15:51:26 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f208f03b000 Jun 24 15:51:26 fir-md1-s1 kernel: LustreError: 23103:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.8.26@o2ib6 arrived at 1561416686 with bad export cookie 6746082289097843395 Jun 24 15:51:28 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f248da4a600 Jun 24 15:51:28 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2356cf4a00 Jun 24 15:51:29 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f16fab25c00 Jun 24 15:51:29 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1632bfc800 Jun 24 15:51:29 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1c5c907a00 Jun 24 15:51:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO read error with 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6), client will retry: rc -110 Jun 24 15:51:30 fir-md1-s1 kernel: LustreError: 20722:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f4505753000 ns: mdt-fir-MDT0002_UUID lock: ffff8f17a141da00/0x5d9ee6233fcc727c lrc: 1/0,0 mode: EX/EX res: [0x2c002bf83:0xe7e7:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x54801000000000 nid: 10.8.29.5@o2ib6 remote: 0xc606c8a810cda319 expref: 12 pid: 20722 timeout: 0 lvb_type: 3 Jun 24 15:51:30 fir-md1-s1 kernel: LustreError: 25074:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.5@o2ib6 arrived at 1561416690 with bad export cookie 6746082289097820148 Jun 24 15:51:30 fir-md1-s1 kernel: LustreError: 20722:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message Jun 24 15:51:31 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f167a83e600 Jun 24 15:51:32 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1cd303b400 Jun 24 15:51:33 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f162cfea000 Jun 24 15:51:33 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1af4245e00 Jun 24 15:51:34 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f162cfe8e00 Jun 24 15:51:34 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f34cb63a800 Jun 24 15:51:34 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1ceab74e00 Jun 24 15:51:34 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1628a03600 Jun 24 15:51:34 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d3aec7800 Jun 24 15:51:35 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1628a07a00 Jun 24 15:51:35 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f179ab17800 Jun 24 15:51:36 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f4490bf9800 Jun 24 15:51:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.24@o2ib6, removing former export from same NID Jun 24 15:51:36 fir-md1-s1 kernel: Lustre: Skipped 724 previous similar messages Jun 24 15:51:36 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1a55fccc00 Jun 24 15:51:36 fir-md1-s1 kernel: LustreError: 22891:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.8.26@o2ib6 arrived at 1561416696 with bad export cookie 6746082289097843395 Jun 24 15:51:36 fir-md1-s1 kernel: LustreError: 22891:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 5 previous similar messages Jun 24 15:51:36 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1da986f800 Jun 24 15:51:38 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f204374ba00 Jun 24 15:51:39 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2b8438ca00 Jun 24 15:51:39 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f33c9620e00 Jun 24 15:51:39 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f24ab328800 Jun 24 15:51:40 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f33c9626000 Jun 24 15:51:40 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d59e1fc00 Jun 24 15:51:41 fir-md1-s1 kernel: LustreError: 20721:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f24fe577800 ns: mdt-fir-MDT0002_UUID lock: ffff8f0946a6bf00/0x5d9ee6233fe3d0a4 lrc: 3/0,0 mode: PW/PW res: [0x2c002bf5a:0x62a9:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x50200000000000 nid: 10.8.17.15@o2ib6 remote: 0x5909d6587625933f expref: 3 pid: 20721 timeout: 0 lvb_type: 0 Jun 24 15:51:41 fir-md1-s1 kernel: LustreError: 20721:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message Jun 24 15:51:41 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f3091ef9c00 Jun 24 15:51:41 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1eeae51c00 Jun 24 15:51:41 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f3a9fefa800 Jun 24 15:51:41 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f167ee3e200 Jun 24 15:51:42 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1fffdaf200 Jun 24 15:51:42 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1f8fb80e00 Jun 24 15:51:42 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f167ee3ee00 Jun 24 15:51:42 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1fffdafe00 Jun 24 15:51:42 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1fffda9200 Jun 24 15:51:42 fir-md1-s1 kernel: LustreError: 21389:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 30+0s req@ffff8f1ee25a5050 x1631564825922512/t0(0) o4->04031d35-e75a-0623-0a2e-3f8a84f80ab5@10.8.27.15@o2ib6:12/0 lens 488/448 e 0 to 0 dl 1561416702 ref 1 fl Interpret:/0/0 rc 0/0 Jun 24 15:51:42 fir-md1-s1 kernel: LustreError: 21389:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 22 previous similar messages Jun 24 15:51:43 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1647e77c00 Jun 24 15:51:43 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f164af5c000 Jun 24 15:51:43 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1ed9e8fe00 Jun 24 15:51:43 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2504f73400 Jun 24 15:51:43 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d7b9b4a00 Jun 24 15:51:43 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f22e4b21000 Jun 24 15:51:45 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f0b3ca2f600 Jun 24 15:51:45 fir-md1-s1 kernel: LustreError: 23101:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.28.11@o2ib6 arrived at 1561416705 with bad export cookie 6746082289092222801 Jun 24 15:51:45 fir-md1-s1 kernel: LustreError: 23101:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 1 previous similar message Jun 24 15:51:48 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f17bb5fb000 Jun 24 15:51:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO read error with b09d4c25-b109-b30c-132e-6a644105be34 (at 10.8.9.9@o2ib6), client will retry: rc -110 Jun 24 15:51:48 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 24 15:51:48 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2536ed6400 Jun 24 15:51:49 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f182d76fe00 Jun 24 15:51:51 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f182d70fe00 Jun 24 15:51:57 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1798378400 Jun 24 15:52:05 fir-md1-s1 kernel: LustreError: 50446:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.10@o2ib6) failed to reply to blocking AST (req@ffff8f1fb8f0c500 x1636713475156992 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f1e163d0b40/0x5d9ee622da3415bb lrc: 4/0,0 mode: PR/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 1234 type: IBT flags: 0x60200400000020 nid: 10.8.9.10@o2ib6 remote: 0x9ed6a5314c69ab45 expref: 766118 pid: 24587 timeout: 531794 lvb_type: 0 Jun 24 15:52:05 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.9.10@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jun 24 15:52:05 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 49s: evicting client at 10.8.9.10@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1e163d0b40/0x5d9ee622da3415bb lrc: 3/0,0 mode: PR/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 1233 type: IBT flags: 0x60200400000020 nid: 10.8.9.10@o2ib6 remote: 0x9ed6a5314c69ab45 expref: 766095 pid: 24587 timeout: 0 lvb_type: 0 Jun 24 15:52:05 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages Jun 24 15:52:05 fir-md1-s1 kernel: LustreError: 20930:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.9.10@o2ib6 arrived at 1561416725 with bad export cookie 6746082289090716541 Jun 24 15:52:05 fir-md1-s1 kernel: LustreError: 20930:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 15 previous similar messages Jun 24 15:52:15 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2305afe200 Jun 24 15:52:15 fir-md1-s1 kernel: Lustre: 21896:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:5s); client may timeout. req@ffff8f24f4914450 x1635709199425856/t0(0) o37->09fe1fc8-d186-6314-b715-72bcbbf4dcb1@10.8.1.35@o2ib6:10/0 lens 448/408 e 1 to 0 dl 1561416730 ref 1 fl Complete:/0/0 rc -110/-110 Jun 24 15:52:15 fir-md1-s1 kernel: Lustre: 21896:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 78 previous similar messages Jun 24 15:52:16 fir-md1-s1 kernel: LustreError: 97641:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.0.65@o2ib6: deadline 30:4s ago req@ffff8f18a2079500 x1634092354836336/t0(0) o101->87da5719-38f8-e25f-27bd-899baebba0f4@10.8.0.65@o2ib6:12/0 lens 576/0 e 0 to 0 dl 1561416732 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jun 24 15:52:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 5d60b790-0b15-ff01-65b5-d8a0250b0e53 (at 10.8.1.29@o2ib6), client will retry: rc = -110 Jun 24 15:52:23 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jun 24 15:52:23 fir-md1-s1 kernel: LustreError: 50444:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.8.12@o2ib6: deadline 30:7s ago req@ffff8f1a72f4cb00 x1634455977415344/t0(0) o101->b95afc0f-d5ce-0d5e-e5e9-03cd8d169d60@10.8.8.12@o2ib6:16/0 lens 576/0 e 0 to 0 dl 1561416736 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jun 24 15:52:23 fir-md1-s1 kernel: LustreError: 50444:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 89 previous similar messages Jun 24 15:52:26 fir-md1-s1 kernel: LustreError: 97666:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.17.9@o2ib6: deadline 30:3s ago req@ffff8f1d20601e00 x1635343772185568/t0(0) o101->51002e48-a06e-3405-fcaa-ac377ed743af@10.8.17.9@o2ib6:23/0 lens 576/0 e 0 to 0 dl 1561416743 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jun 24 15:52:26 fir-md1-s1 kernel: LustreError: 97666:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 474 previous similar messages Jun 24 15:52:46 fir-md1-s1 kernel: LustreError: 10197:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561416676, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f13baa669c0/0x5d9ee6233fef45b2 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10197 timeout: 0 lvb_type: 0 Jun 24 15:52:46 fir-md1-s1 kernel: LustreError: 10197:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 25 previous similar messages Jun 24 15:52:55 fir-md1-s1 kernel: LustreError: 23582:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561416684, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f3c12216780/0x5d9ee6233fefc9f5 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23582 timeout: 0 lvb_type: 0 Jun 24 15:52:55 fir-md1-s1 kernel: LustreError: 23582:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 148 previous similar messages Jun 24 15:53:10 fir-md1-s1 kernel: LustreError: 21415:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.106.17@o2ib4: deadline 30:1s ago req@ffff8f2f45f40c00 x1634122400223376/t0(0) o101->459a4674-896d-e57f-5fbe-6e6932e88880@10.9.106.17@o2ib4:9/0 lens 576/0 e 0 to 0 dl 1561416789 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jun 24 15:53:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.10@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 24 15:53:10 fir-md1-s1 kernel: LustreError: 21415:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 210 previous similar messages Jun 24 15:53:14 fir-md1-s1 kernel: LustreError: 22282:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f162a623c00 x1636713475360464/t0(0) o104->fir-MDT0002@10.8.27.35@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 24 15:53:14 fir-md1-s1 kernel: LustreError: 22282:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jun 24 15:53:30 fir-md1-s1 kernel: LustreError: 20462:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f18dbffb000 x1636713475556288/t0(0) o104->fir-MDT0000@10.8.9.10@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 24 15:53:39 fir-md1-s1 kernel: Lustre: 20731:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1d5876e000 x1636443218517136/t0(0) o101->7b7e9b9d-7d80-a5c4-07fd-dd92cbcbe2f0@10.8.29.6@o2ib6:14/0 lens 1784/3288 e 0 to 0 dl 1561416824 ref 2 fl Interpret:/0/0 rc 0/0 Jun 24 15:53:39 fir-md1-s1 kernel: Lustre: 20731:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2191 previous similar messages Jun 24 15:53:43 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.27.35@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2356b0d580/0x5d9ee62327c3cc87 lrc: 3/0,0 mode: PR/PR res: [0x2c002be88:0xe04f:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.27.35@o2ib6 remote: 0xe7fd3d175f79dfa5 expref: 71235 pid: 21481 timeout: 531883 lvb_type: 0 Jun 24 15:54:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to bdf06334-3a1e-8f45-20cb-38a64ac80139 (at 10.8.29.5@o2ib6) Jun 24 15:54:32 fir-md1-s1 kernel: Lustre: Skipped 2498 previous similar messages Jun 24 15:55:00 fir-md1-s1 kernel: LustreError: 20726:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561416810, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1ee5f38240/0x5d9ee6234033997e lrc: 3/0,1 mode: --/PW res: [0x2000222aa:0x10e:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20726 timeout: 0 lvb_type: 0 Jun 24 15:55:00 fir-md1-s1 kernel: LustreError: 20462:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561416810, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2f1ab94380/0x5d9ee62340339970 lrc: 3/0,1 mode: --/PW res: [0x200025b09:0x2437:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20462 timeout: 0 lvb_type: 0 Jun 24 15:55:00 fir-md1-s1 kernel: LustreError: 20462:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 216 previous similar messages Jun 24 15:55:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b37c54be-7fed-724b-d760-c5bd71b2a4e0 (at 10.8.29.5@o2ib6) reconnecting Jun 24 15:55:03 fir-md1-s1 kernel: Lustre: Skipped 998 previous similar messages Jun 24 15:57:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) in 230 seconds. I think it's dead, and I am evicting it. exp ffff8f22f15df000, cur 1561417020 expire 1561416870 last 1561416790 Jun 24 16:14:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) Jun 24 16:14:43 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 24 21:20:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ffa27290-6cf4-9b77-ab2a-7df1aa693fad (at 10.8.21.21@o2ib6) Jun 24 21:21:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ae85bd6d-3abb-15dd-50c5-ec36d3fe0421 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1867500400, cur 1561436462 expire 1561436312 last 1561436235 Jun 24 22:52:55 fir-md1-s1 kernel: LNetError: 20192:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jun 24 22:52:55 fir-md1-s1 kernel: LNetError: 20192:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 88 previous similar messages Jun 25 09:03:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d4a6325e-22ba-0473-b0bb-1ac629cc9b52 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2501b00c00, cur 1561478620 expire 1561478470 last 1561478393 Jun 25 09:03:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 09:03:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jun 25 09:03:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 09:07:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jun 25 09:07:33 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 2ae4f990-b2cb-626b-12c1-a51b5888422d (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f28e523c400, cur 1561478853 expire 1561478703 last 1561478626 Jun 25 09:07:33 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 12:40:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 43eb156b-cf2c-6d44-b021-842e2a3ba6bf (at 10.8.14.1@o2ib6) Jun 25 12:40:58 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 12:41:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 2900ab2e-d5c8-984c-4497-834ead5e0c0c (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f3ea0000, cur 1561491675 expire 1561491525 last 1561491448 Jun 25 12:41:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 2900ab2e-d5c8-984c-4497-834ead5e0c0c (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f252023b800, cur 1561491686 expire 1561491536 last 1561491459 Jun 25 12:41:26 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 12:53:37 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 25 12:53:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) reconnecting Jun 25 12:53:44 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 12:53:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jun 25 12:53:44 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 13:37:48 fir-md1-s1 kernel: Lustre: 21370:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f0b9d97b600 x1634476177653344/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:23/0 lens 480/568 e 0 to 0 dl 1561495073 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 13:37:48 fir-md1-s1 kernel: Lustre: 21370:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Jun 25 13:37:52 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.8.31@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f3164e28480/0x5d9ee62524f94c87 lrc: 3/0,0 mode: PW/PW res: [0x2c0001757:0xc13:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.8.8.31@o2ib6 remote: 0x4d059c3e6f4a8b90 expref: 23720 pid: 23454 timeout: 610132 lvb_type: 0 Jun 25 13:37:52 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages Jun 25 13:37:53 fir-md1-s1 kernel: LustreError: 20369:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.8.31@o2ib6 arrived at 1561495073 with bad export cookie 6746082289096703970 Jun 25 13:37:53 fir-md1-s1 kernel: LustreError: 20369:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 5917 previous similar messages Jun 25 13:37:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 8172217c-cb28-d209-5f1f-4aceb1d4d3a6 (at 10.8.8.31@o2ib6) Jun 25 13:37:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 13:37:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jun 25 13:40:06 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14c7771c00, cur 1561495206 expire 1561495056 last 1561494979 Jun 25 13:40:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4537985c00, cur 1561495219 expire 1561495069 last 1561494992 Jun 25 13:41:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e18301fc-f860-0db4-bf24-6c606e0cc839 (at 10.8.8.31@o2ib6) in 222 seconds. I think it's dead, and I am evicting it. exp ffff8f162e7b6800, cur 1561495295 expire 1561495145 last 1561495073 Jun 25 13:41:36 fir-md1-s1 kernel: Lustre: 20720:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1a80bd7200 x1631309824666512/t0(0) o101->2defae61-8bf0-dee6-7d48-53b83a69e973@10.8.17.24@o2ib6:11/0 lens 480/568 e 0 to 0 dl 1561495301 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 13:41:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 8172217c-cb28-d209-5f1f-4aceb1d4d3a6 (at 10.8.8.31@o2ib6) Jun 25 13:41:40 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.7.8@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f20c534ad00/0x5d9ee62524f5a84c lrc: 3/0,0 mode: PW/PW res: [0x2c002c286:0x916e:0x0].0x0 bits 0x40/0x0 rrc: 24 type: IBT flags: 0x60200400000020 nid: 10.8.7.8@o2ib6 remote: 0x9a03b0d8ce0febf6 expref: 351 pid: 97651 timeout: 610360 lvb_type: 0 Jun 25 13:41:43 fir-md1-s1 kernel: LustreError: 25084:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.7.8@o2ib6 arrived at 1561495303 with bad export cookie 6746082289090927066 Jun 25 13:41:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to d5145b19-7e77-2465-cb06-19cf549382e1 (at 10.8.7.8@o2ib6) Jun 25 13:41:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f3e5ac00, cur 1561495315 expire 1561495165 last 1561495088 Jun 25 13:46:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client f8fcb29b-d706-0b08-6893-aa94c8d5e667 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f233e810000, cur 1561495577 expire 1561495427 last 1561495350 Jun 25 13:46:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jun 25 13:46:57 fir-md1-s1 kernel: Lustre: 23622:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f351a0f7850 x1634476184772000/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:2/0 lens 480/568 e 1 to 0 dl 1561495622 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 13:47:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 13:47:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jun 25 13:47:03 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 13:47:08 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.17.15@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1cff79f740/0x5d9ee62527f6acb5 lrc: 3/0,0 mode: PW/PW res: [0x2c002c286:0x915a:0x0].0x0 bits 0x40/0x0 rrc: 20 type: IBT flags: 0x60200400000020 nid: 10.8.17.15@o2ib6 remote: 0x5909d658763498c3 expref: 741 pid: 97643 timeout: 610688 lvb_type: 0 Jun 25 13:47:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 13:47:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jun 25 13:47:24 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 25 13:48:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) Jun 25 13:48:10 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 13:48:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) reconnecting Jun 25 13:48:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 13:48:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 13:48:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID Jun 25 13:48:41 fir-md1-s1 kernel: Lustre: Skipped 461 previous similar messages Jun 25 14:06:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jun 25 14:06:53 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 25 14:06:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d5fc548e-054d-12d9-54b9-977767ad7c03 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f16536ff400, cur 1561496817 expire 1561496667 last 1561496590 Jun 25 14:06:57 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:12:50 fir-md1-s1 kernel: Lustre: 23595:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561497163/real 1561497163] req@ffff8f1073b68300 x1636714409925136/t0(0) o104->fir-MDT0002@10.8.14.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561497170 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 25 14:12:50 fir-md1-s1 kernel: Lustre: 23595:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 534 previous similar messages Jun 25 14:12:58 fir-md1-s1 kernel: Lustre: 23598:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f07c3ec0c00 x1631601318285360/t0(0) o101->f1b26272-cb99-9dbe-fdc3-6a70f1d77cbb@10.9.112.4@o2ib4:3/0 lens 1784/3288 e 1 to 0 dl 1561497183 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 14:12:58 fir-md1-s1 kernel: Lustre: 23598:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jun 25 14:13:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client f1b26272-cb99-9dbe-fdc3-6a70f1d77cbb (at 10.9.112.4@o2ib4) reconnecting Jun 25 14:13:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 14:13:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 4ac555d7-5727-5203-83f8-102dd77ed0e4 (at 10.9.112.4@o2ib4) Jun 25 14:13:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:13:11 fir-md1-s1 kernel: Lustre: 23595:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561497184/real 1561497184] req@ffff8f1073b68300 x1636714409925136/t0(0) o104->fir-MDT0002@10.8.14.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561497191 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 14:13:11 fir-md1-s1 kernel: Lustre: 23595:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jun 25 14:13:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client f1b26272-cb99-9dbe-fdc3-6a70f1d77cbb (at 10.9.112.4@o2ib4) reconnecting Jun 25 14:13:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:13:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 4ac555d7-5727-5203-83f8-102dd77ed0e4 (at 10.9.112.4@o2ib4) Jun 25 14:13:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:13:46 fir-md1-s1 kernel: Lustre: 23595:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561497219/real 1561497219] req@ffff8f1073b68300 x1636714409925136/t0(0) o104->fir-MDT0002@10.8.14.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561497226 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 14:13:46 fir-md1-s1 kernel: Lustre: 23595:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jun 25 14:13:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client f1b26272-cb99-9dbe-fdc3-6a70f1d77cbb (at 10.9.112.4@o2ib4) reconnecting Jun 25 14:13:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:14:07 fir-md1-s1 kernel: Lustre: 10149:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f161ea8c200 x1631676600998688/t0(0) o101->92ffa420-d747-a973-baf2-68cec64e7e81@10.9.113.14@o2ib4:12/0 lens 1784/3288 e 1 to 0 dl 1561497252 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 14:14:07 fir-md1-s1 kernel: Lustre: 10149:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jun 25 14:14:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 4ac555d7-5727-5203-83f8-102dd77ed0e4 (at 10.9.112.4@o2ib4) Jun 25 14:14:07 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jun 25 14:14:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client f1b26272-cb99-9dbe-fdc3-6a70f1d77cbb (at 10.9.112.4@o2ib4) reconnecting Jun 25 14:14:28 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jun 25 14:14:36 fir-md1-s1 kernel: LustreError: 23595:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.14.1@o2ib6) returned error from blocking AST (req@ffff8f1073b68300 x1636714409925136 status -107 rc -107), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f197b5bd580/0x5d9ee6251a154b9d lrc: 4/0,0 mode: PR/PR res: [0x2c002bea6:0x1eebd:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.14.1@o2ib6 remote: 0xd0d7046257247141 expref: 712 pid: 97641 timeout: 612484 lvb_type: 0 Jun 25 14:14:36 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.14.1@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Jun 25 14:14:36 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 113s: evicting client at 10.8.14.1@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1f5065c800/0x5d9ee6251a155903 lrc: 3/0,0 mode: PR/PR res: [0x2c002bea6:0x1ee30:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.14.1@o2ib6 remote: 0xd0d7046257247466 expref: 713 pid: 20545 timeout: 0 lvb_type: 0 Jun 25 14:14:36 fir-md1-s1 kernel: LustreError: 23595:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Jun 25 14:15:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 43eb156b-cf2c-6d44-b021-842e2a3ba6bf (at 10.8.14.1@o2ib6) Jun 25 14:15:51 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jun 25 14:15:57 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client aa5f6715-716f-cf30-713a-acb85093703e (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2501639400, cur 1561497357 expire 1561497207 last 1561497130 Jun 25 14:15:57 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:20:14 fir-md1-s1 kernel: Lustre: 20555:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1af56cad00 x1634476201059968/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:19/0 lens 480/568 e 0 to 0 dl 1561497619 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 14:20:18 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.8.31@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2521359680/0x5d9ee62534abbefd lrc: 3/0,0 mode: PW/PW res: [0x2c0001757:0xc13:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.8.8.31@o2ib6 remote: 0x4d059c3e6f4b4086 expref: 23 pid: 20722 timeout: 612678 lvb_type: 0 Jun 25 14:20:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 8172217c-cb28-d209-5f1f-4aceb1d4d3a6 (at 10.8.8.31@o2ib6) Jun 25 14:20:36 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:22:23 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client e4594a87-2fe5-1bf8-dbe3-26a702178742 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24e7dd9800, cur 1561497743 expire 1561497593 last 1561497516 Jun 25 14:22:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 14:24:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ebef6758-802b-3d88-0fb7-39f9e3a97c72 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2521152400, cur 1561497853 expire 1561497703 last 1561497626 Jun 25 14:26:30 fir-md1-s1 kernel: Lustre: 21668:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f148a6a8c00 x1634476203917216/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:5/0 lens 480/568 e 0 to 0 dl 1561497995 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 14:26:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 14:26:37 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:26:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jun 25 14:26:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.68@o2ib6, removing former export from same NID Jun 25 14:26:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 14:26:54 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 25 14:27:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 14:27:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 14:27:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.68@o2ib6, removing former export from same NID Jun 25 14:27:36 fir-md1-s1 kernel: LustreError: 23653:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561497965, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f153468c380/0x5d9ee62537843d97 lrc: 3/0,1 mode: --/PW res: [0x2c0001757:0xc13:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23653 timeout: 0 lvb_type: 0 Jun 25 14:27:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 14:27:39 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 14:28:35 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.8.8.31@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2059d557c0/0x5d9ee625376e5444 lrc: 3/0,0 mode: PW/PW res: [0x2c0001757:0xc13:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.8.31@o2ib6 remote: 0x4d059c3e6f4b4fcf expref: 22 pid: 22007 timeout: 613175 lvb_type: 0 Jun 25 14:29:47 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f10667e1800, cur 1561498187 expire 1561498037 last 1561497960 Jun 25 14:29:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 14:31:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25218f0400, cur 1561498278 expire 1561498128 last 1561498051 Jun 25 14:35:48 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561498537/real 1561498537] req@ffff8f22c1417200 x1636714432316944/t0(0) o104->fir-MDT0002@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561498548 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 25 14:35:48 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 34 previous similar messages Jun 25 14:35:55 fir-md1-s1 kernel: Lustre: 25678:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0d7c3b0f00 x1634123299186224/t0(0) o36->6c224fde-2a1b-f3eb-fdf9-6a986a61a55a@10.9.108.4@o2ib4:0/0 lens 536/2888 e 1 to 0 dl 1561498560 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 14:35:59 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561498548/real 1561498548] req@ffff8f22c1417200 x1636714432316944/t0(0) o104->fir-MDT0002@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561498559 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 14:36:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6c224fde-2a1b-f3eb-fdf9-6a986a61a55a (at 10.9.108.4@o2ib4) reconnecting Jun 25 14:36:01 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 14:36:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to cffa9ca6-4860-be91-20b9-abd21a031d37 (at 10.9.108.4@o2ib4) Jun 25 14:36:01 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jun 25 14:36:21 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561498570/real 1561498570] req@ffff8f22c1417200 x1636714432316944/t0(0) o104->fir-MDT0002@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561498581 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 14:36:21 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 25 14:36:54 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561498603/real 1561498603] req@ffff8f22c1417200 x1636714432316944/t0(0) o104->fir-MDT0002@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561498614 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 14:36:54 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 25 14:37:02 fir-md1-s1 kernel: LustreError: 23716:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561498532, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1f0e79cc80/0x5d9ee62539efccf5 lrc: 3/0,1 mode: --/PW res: [0x2c0001757:0xc13:0x0].0x0 bits 0x40/0x0 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23716 timeout: 0 lvb_type: 0 Jun 25 14:37:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID Jun 25 14:37:10 fir-md1-s1 kernel: Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Jun 25 14:37:10 fir-md1-s1 kernel: LustreError: 21370:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561498540, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8f10d804cc80/0x5d9ee62539f7e3a5 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x1000001000000 nid: local remote: 0x5d9ee62539f7e3ac expref: -99 pid: 21370 timeout: 0 lvb_type: 0 Jun 25 14:37:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) reconnecting Jun 25 14:37:10 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jun 25 14:37:28 fir-md1-s1 kernel: LustreError: 97658:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561498558, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8f184bf06300/0x5d9ee6253a1036b8 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x1000001000000 nid: local remote: 0x5d9ee6253a1036bf expref: -99 pid: 97658 timeout: 0 lvb_type: 0 Jun 25 14:37:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID Jun 25 14:37:36 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 14:38:00 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561498669/real 1561498669] req@ffff8f22c1417200 x1636714432316944/t0(0) o104->fir-MDT0002@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561498680 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 14:38:00 fir-md1-s1 kernel: Lustre: 20555:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jun 25 14:38:01 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.8.8.31@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1f699f9680/0x5d9ee62539ecd807 lrc: 3/0,0 mode: PW/PW res: [0x2c0001757:0xc13:0x0].0x0 bits 0x40/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.8.31@o2ib6 remote: 0x4d059c3e6f4b6531 expref: 20 pid: 50445 timeout: 613741 lvb_type: 0 Jun 25 14:38:11 fir-md1-s1 kernel: LustreError: 20555:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.0.65@o2ib6) failed to reply to blocking AST (req@ffff8f22c1417200 x1636714432316944 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f22fcc47740/0x5d9ee6252ea5cf70 lrc: 4/0,0 mode: PR/PR res: [0x2c0000404:0x2d3:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.0.65@o2ib6 remote: 0xf3ad1a144a9c4e3 expref: 749697 pid: 23455 timeout: 613889 lvb_type: 0 Jun 25 14:38:11 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.0.65@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jun 25 14:38:11 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 25 14:38:12 fir-md1-s1 kernel: LustreError: 25086:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498692 with bad export cookie 6746082339115562538 Jun 25 14:38:13 fir-md1-s1 kernel: LustreError: 25086:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498693 with bad export cookie 6746082339115562538 Jun 25 14:38:13 fir-md1-s1 kernel: LustreError: 25086:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 9 previous similar messages Jun 25 14:38:15 fir-md1-s1 kernel: LustreError: 25086:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498695 with bad export cookie 6746082339115562538 Jun 25 14:38:15 fir-md1-s1 kernel: LustreError: 25086:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 11 previous similar messages Jun 25 14:38:19 fir-md1-s1 kernel: LustreError: 25078:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498699 with bad export cookie 6746082339115562538 Jun 25 14:38:19 fir-md1-s1 kernel: LustreError: 25078:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 19 previous similar messages Jun 25 14:38:28 fir-md1-s1 kernel: LustreError: 25029:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498708 with bad export cookie 6746082339115562538 Jun 25 14:38:28 fir-md1-s1 kernel: LustreError: 25029:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 29 previous similar messages Jun 25 14:38:44 fir-md1-s1 kernel: LustreError: 25029:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498724 with bad export cookie 6746082339115562538 Jun 25 14:38:44 fir-md1-s1 kernel: LustreError: 25029:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 55 previous similar messages Jun 25 14:38:57 fir-md1-s1 kernel: LNet: Service thread pid 20555 was inactive for 200.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 25 14:38:57 fir-md1-s1 kernel: Pid: 20555, comm: mdt01_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 25 14:38:57 fir-md1-s1 kernel: Call Trace: Jun 25 14:38:57 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x890 [ptlrpc] Jun 25 14:38:57 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 25 14:38:57 fir-md1-s1 kernel: [] mdt_object_local_lock+0x438/0xb20 [mdt] Jun 25 14:38:57 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 25 14:38:57 fir-md1-s1 kernel: [] mdt_reint_object_lock+0x2c/0x60 [mdt] Jun 25 14:38:57 fir-md1-s1 kernel: [] mdt_object_lock_save+0x29/0x50 [mdt] Jun 25 14:38:57 fir-md1-s1 kernel: [] mdt_reint_rename+0x4ce/0x2b90 [mdt] Jun 25 14:38:57 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jun 25 14:38:57 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jun 25 14:38:57 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jun 25 14:38:57 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 25 14:38:57 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 25 14:38:57 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 25 14:38:57 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 25 14:38:57 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 25 14:38:57 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 25 14:38:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561498737.20555 Jun 25 14:38:58 fir-md1-s1 kernel: LNet: Service thread pid 50445 was inactive for 200.31s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 25 14:38:58 fir-md1-s1 kernel: Pid: 50445, comm: mdt01_073 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 25 14:38:58 fir-md1-s1 kernel: Call Trace: Jun 25 14:38:58 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 25 14:38:58 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 25 14:38:58 fir-md1-s1 kernel: [] mdt_rename_lock+0x24b/0x4b0 [mdt] Jun 25 14:38:58 fir-md1-s1 kernel: [] mdt_reint_rename+0x2c5/0x2b90 [mdt] Jun 25 14:38:58 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jun 25 14:38:58 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jun 25 14:38:58 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jun 25 14:38:58 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 25 14:38:58 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 25 14:38:58 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 25 14:38:58 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 25 14:38:58 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 25 14:38:58 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 25 14:38:58 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561498738.50445 Jun 25 14:39:00 fir-md1-s1 kernel: LNet: Service thread pid 21370 was inactive for 200.25s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 25 14:39:00 fir-md1-s1 kernel: Pid: 21370, comm: mdt00_012 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 25 14:39:00 fir-md1-s1 kernel: Call Trace: Jun 25 14:39:00 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 25 14:39:00 fir-md1-s1 kernel: [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] Jun 25 14:39:00 fir-md1-s1 kernel: [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] Jun 25 14:39:00 fir-md1-s1 kernel: [] osp_md_object_lock+0x162/0x2d0 [osp] Jun 25 14:39:00 fir-md1-s1 kernel: [] lod_object_lock+0xf3/0x7b0 [lod] Jun 25 14:39:00 fir-md1-s1 kernel: [] mdd_object_lock+0x3e/0xe0 [mdd] Jun 25 14:39:00 fir-md1-s1 kernel: [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] Jun 25 14:39:00 fir-md1-s1 kernel: [] mdt_remote_object_lock+0x2a/0x30 [mdt] Jun 25 14:39:00 fir-md1-s1 kernel: [] mdt_rename_lock+0xbe/0x4b0 [mdt] Jun 25 14:39:00 fir-md1-s1 kernel: [] mdt_reint_rename+0x2c5/0x2b90 [mdt] Jun 25 14:39:00 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jun 25 14:39:00 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jun 25 14:39:00 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jun 25 14:39:00 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 25 14:39:00 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 25 14:39:00 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 25 14:39:00 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 25 14:39:00 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 25 14:39:00 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 25 14:39:00 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561498740.21370 Jun 25 14:39:16 fir-md1-s1 kernel: LustreError: 22009:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498756 with bad export cookie 6746082339115562538 Jun 25 14:39:16 fir-md1-s1 kernel: LustreError: 22009:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 112 previous similar messages Jun 25 14:39:19 fir-md1-s1 kernel: LNet: Service thread pid 97658 was inactive for 200.48s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 25 14:39:19 fir-md1-s1 kernel: Pid: 97658, comm: mdt01_097 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 25 14:39:19 fir-md1-s1 kernel: Call Trace: Jun 25 14:39:19 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 25 14:39:19 fir-md1-s1 kernel: [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] Jun 25 14:39:19 fir-md1-s1 kernel: [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] Jun 25 14:39:19 fir-md1-s1 kernel: [] osp_md_object_lock+0x162/0x2d0 [osp] Jun 25 14:39:19 fir-md1-s1 kernel: [] lod_object_lock+0xf3/0x7b0 [lod] Jun 25 14:39:19 fir-md1-s1 kernel: [] mdd_object_lock+0x3e/0xe0 [mdd] Jun 25 14:39:19 fir-md1-s1 kernel: [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] Jun 25 14:39:19 fir-md1-s1 kernel: [] mdt_remote_object_lock+0x2a/0x30 [mdt] Jun 25 14:39:19 fir-md1-s1 kernel: [] mdt_rename_lock+0xbe/0x4b0 [mdt] Jun 25 14:39:19 fir-md1-s1 kernel: [] mdt_reint_rename+0x2c5/0x2b90 [mdt] Jun 25 14:39:19 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jun 25 14:39:19 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jun 25 14:39:19 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jun 25 14:39:19 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 25 14:39:19 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 25 14:39:19 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 25 14:39:19 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 25 14:39:19 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 25 14:39:19 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 25 14:39:19 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561498759.97658 Jun 25 14:39:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6c224fde-2a1b-f3eb-fdf9-6a986a61a55a (at 10.9.108.4@o2ib4) reconnecting Jun 25 14:39:31 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jun 25 14:39:41 fir-md1-s1 kernel: LustreError: 20555:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561498691, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f2517b7d7c0/0x5d9ee62539f50611 lrc: 3/0,1 mode: --/CW res: [0x2c0000404:0x2d3:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20555 timeout: 0 lvb_type: 0 Jun 25 14:39:41 fir-md1-s1 kernel: LustreError: 20555:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jun 25 14:40:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID Jun 25 14:40:20 fir-md1-s1 kernel: LustreError: 25080:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498820 with bad export cookie 6746082339115562538 Jun 25 14:40:20 fir-md1-s1 kernel: LustreError: 25080:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 135 previous similar messages Jun 25 14:40:32 fir-md1-s1 kernel: LNet: Service thread pid 20555 completed after 295.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 25 14:40:32 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Jun 25 14:42:38 fir-md1-s1 kernel: LustreError: 25030:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561498958 with bad export cookie 6746082339115562538 Jun 25 14:42:38 fir-md1-s1 kernel: LustreError: 25030:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 333 previous similar messages Jun 25 14:46:55 fir-md1-s1 kernel: LustreError: 22891:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561499215 with bad export cookie 6746082339115562538 Jun 25 14:46:55 fir-md1-s1 kernel: LustreError: 22891:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 559 previous similar messages Jun 25 14:48:11 fir-md1-s1 kernel: Lustre: 10149:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f339c6e1800 x1634476211139808/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:16/0 lens 480/568 e 1 to 0 dl 1561499296 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 14:48:11 fir-md1-s1 kernel: Lustre: 10149:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Jun 25 14:48:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 14:48:17 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jun 25 14:48:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jun 25 14:48:17 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jun 25 14:48:52 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.18.30@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f0aa8b733c0/0x5d9ee6253d84ebe9 lrc: 3/0,0 mode: PW/PW res: [0x2c002c286:0x22f9:0x0].0x0 bits 0x40/0x0 rrc: 21 type: IBT flags: 0x60200400000020 nid: 10.8.18.30@o2ib6 remote: 0xb6237814b733a7c0 expref: 347 pid: 21455 timeout: 614392 lvb_type: 0 Jun 25 14:48:52 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 25 14:52:22 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.29.8@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f0cd2226540/0x5d9ee6253e89b2e2 lrc: 3/0,0 mode: PW/PW res: [0x200025b67:0x15cdc:0x0].0x0 bits 0x40/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.29.8@o2ib6 remote: 0xfccaff921072149f expref: 182 pid: 97658 timeout: 614602 lvb_type: 0 Jun 25 14:52:22 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Jun 25 14:52:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 7b8c2334-5441-fafb-761f-7bfdc2fe1e61 (at 10.8.18.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1de3da3800, cur 1561499578 expire 1561499428 last 1561499351 Jun 25 14:54:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client b09d4c25-b109-b30c-132e-6a644105be34 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d4620bc00, cur 1561499675 expire 1561499525 last 1561499448 Jun 25 14:54:35 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 14:55:44 fir-md1-s1 kernel: LustreError: 21411:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561499654, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f2b72f9bcc0/0x5d9ee625401b8723 lrc: 3/0,1 mode: --/PW res: [0x2c0001757:0xc13:0x0].0x0 bits 0x40/0x0 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21411 timeout: 0 lvb_type: 0 Jun 25 14:56:00 fir-md1-s1 kernel: LustreError: 20555:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561499670, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1e95469440/0x5d9ee6254034ca19 lrc: 3/0,1 mode: --/PW res: [0x2c002c286:0x22e6:0x0].0x0 bits 0x40/0x0 rrc: 29 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20555 timeout: 0 lvb_type: 0 Jun 25 14:56:43 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 133s: evicting client at 10.8.8.22@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f445a522400/0x5d9ee6253feda489 lrc: 3/0,0 mode: PW/PW res: [0x2c002c286:0x22e6:0x0].0x0 bits 0x40/0x0 rrc: 29 type: IBT flags: 0x60200400000020 nid: 10.8.8.22@o2ib6 remote: 0xadd31f4354a7f69f expref: 38637 pid: 97658 timeout: 614759 lvb_type: 0 Jun 25 14:56:43 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 25 14:57:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c93f154e-4163-fb6e-f3cf-dea798de7b5a (at 10.8.27.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f19ae21fc00, cur 1561499850 expire 1561499700 last 1561499623 Jun 25 14:59:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) Jun 25 14:59:08 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jun 25 15:00:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e0767d77-866c-9038-3794-0af657e399d1 (at 10.8.8.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2055293c00, cur 1561500034 expire 1561499884 last 1561499807 Jun 25 15:00:34 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 15:02:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f20083d7800, cur 1561500175 expire 1561500025 last 1561499948 Jun 25 15:02:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 15:02:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 15:03:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID Jun 25 15:04:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID Jun 25 15:04:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) reconnecting Jun 25 15:04:27 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jun 25 15:05:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID Jun 25 15:06:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID Jun 25 15:15:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 15:15:53 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 25 15:16:33 fir-md1-s1 kernel: Lustre: 23682:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3a78389b00 x1634476260813312/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:8/0 lens 480/568 e 1 to 0 dl 1561500998 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 15:16:33 fir-md1-s1 kernel: Lustre: 23682:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages Jun 25 15:16:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 15:16:39 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 25 15:16:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jun 25 15:16:39 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jun 25 15:16:47 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.8.31@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f24243abcc0/0x5d9ee6254872031f lrc: 3/0,0 mode: PW/PW res: [0x2c0001757:0xc13:0x0].0x0 bits 0x40/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.8.31@o2ib6 remote: 0x4d059c3e6f4bca95 expref: 33 pid: 20462 timeout: 616067 lvb_type: 0 Jun 25 15:16:47 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 25 15:17:16 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2520809c00, cur 1561501036 expire 1561500886 last 1561500809 Jun 25 15:18:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f251348bc00, cur 1561501091 expire 1561500941 last 1561500864 Jun 25 15:20:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e18301fc-f860-0db4-bf24-6c606e0cc839 (at 10.8.8.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f161d69c400, cur 1561501250 expire 1561501100 last 1561501023 Jun 25 15:20:50 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 15:28:46 fir-md1-s1 kernel: Lustre: 22289:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f213edc9e00 x1636708518966384/t0(0) o101->1b90433c-235e-7531-cfe6-8ebc9f785a9b@10.9.0.64@o2ib4:21/0 lens 480/568 e 1 to 0 dl 1561501731 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 15:28:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 25 15:28:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 25 15:28:52 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 15:29:00 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.9.0.62@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f17534018c0/0x5d9ee6254bde111d lrc: 3/0,0 mode: PW/PW res: [0x2c002bf03:0x6557:0x0].0x0 bits 0x40/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.9.0.62@o2ib4 remote: 0x33d88ec184ad2d05 expref: 210 pid: 27315 timeout: 616800 lvb_type: 0 Jun 25 15:32:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f29d0b45c00, cur 1561501968 expire 1561501818 last 1561501741 Jun 25 15:37:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.0.64@o2ib4, removing former export from same NID Jun 25 15:37:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 15:37:34 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 15:38:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.0.64@o2ib4, removing former export from same NID Jun 25 15:40:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 25 15:40:59 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jun 25 15:40:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 25 15:40:59 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jun 25 15:41:01 fir-md1-s1 kernel: Lustre: 97651:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1e68e5cb00 x1634476271081664/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:6/0 lens 480/568 e 1 to 0 dl 1561502466 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 15:41:15 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.9.102.21@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f44e4c48900/0x5d9ee6254ea4af69 lrc: 3/0,0 mode: PW/PW res: [0x2c002be60:0x9d6:0x0].0x0 bits 0x40/0x0 rrc: 13 type: IBT flags: 0x60200400000020 nid: 10.9.102.21@o2ib4 remote: 0xa578ba1cd90dc370 expref: 338 pid: 10362 timeout: 617535 lvb_type: 0 Jun 25 15:41:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 15:42:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.0.64@o2ib4, removing former export from same NID Jun 25 15:42:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 15:47:20 fir-md1-s1 kernel: Lustre: 20213:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561502239/real 1561502239] req@ffff8f12992d7800 x1636714437752368/t0(0) o6->fir-OST0020-osc-MDT0002@10.0.10.105@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1561502840 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 25 15:47:20 fir-md1-s1 kernel: Lustre: 20213:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 25 15:47:20 fir-md1-s1 kernel: Lustre: fir-OST0020-osc-MDT0002: Connection to fir-OST0020 (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Jun 25 15:57:21 fir-md1-s1 kernel: Lustre: 20213:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561502840/real 1561502840] req@ffff8f12992d7800 x1636714437752368/t0(0) o6->fir-OST0020-osc-MDT0002@10.0.10.105@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1561503441 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 15:57:21 fir-md1-s1 kernel: Lustre: fir-OST0020-osc-MDT0002: Connection to fir-OST0020 (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Jun 25 15:57:21 fir-md1-s1 kernel: Lustre: fir-OST0020-osc-MDT0002: Connection restored to 10.0.10.105@o2ib7 (at 10.0.10.105@o2ib7) Jun 25 15:57:21 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jun 25 16:01:02 fir-md1-s1 kernel: Lustre: 23699:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f44bb01c800 x1636708651115424/t0(0) o101->1b90433c-235e-7531-cfe6-8ebc9f785a9b@10.9.0.64@o2ib4:7/0 lens 480/568 e 1 to 0 dl 1561503667 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 16:01:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 25 16:01:08 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jun 25 16:02:17 fir-md1-s1 kernel: LustreError: 25677:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561503647, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1b29e0e780/0x5d9ee62555b9bb86 lrc: 3/1,0 mode: --/PR res: [0x200021916:0x3ef:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 25677 timeout: 0 lvb_type: 0 Jun 25 16:03:16 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.9.105.69@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff8f16cabf33c0/0x5d9ee6255411ae28 lrc: 3/0,0 mode: PW/PW res: [0x200021916:0x3ef:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.9.105.69@o2ib4 remote: 0xe87422714ee72752 expref: 69 pid: 23715 timeout: 618856 lvb_type: 0 Jun 25 16:07:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 095971d4-2c15-c9c6-8336-964f67ec504b (at 10.9.105.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3a27ab3400, cur 1561504045 expire 1561503895 last 1561503818 Jun 25 16:08:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) in 161 seconds. I think it's dead, and I am evicting it. exp ffff8f22f3f6e800, cur 1561504121 expire 1561503971 last 1561503960 Jun 25 16:09:44 fir-md1-s1 kernel: Lustre: 23555:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0a74ab8c00 x1636708656626272/t0(0) o101->1b90433c-235e-7531-cfe6-8ebc9f785a9b@10.9.0.64@o2ib4:19/0 lens 480/568 e 1 to 0 dl 1561504189 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 16:09:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2522539800, cur 1561504187 expire 1561504037 last 1561503960 Jun 25 16:09:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 25 16:09:50 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jun 25 16:09:58 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.9.101.58@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f11a0ba4c80/0x5d9ee6254ed8ce2d lrc: 3/0,0 mode: PW/PW res: [0x2c002c0cb:0x5289:0x0].0x0 bits 0x40/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 10.9.101.58@o2ib4 remote: 0xa3260ff1ba69df1c expref: 1943 pid: 23738 timeout: 619258 lvb_type: 0 Jun 25 16:10:07 fir-md1-s1 kernel: LustreError: 31007:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.58@o2ib4 arrived at 1561504207 with bad export cookie 6746082289091244866 Jun 25 16:10:07 fir-md1-s1 kernel: LustreError: 31007:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 13405 previous similar messages Jun 25 16:13:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client f29e1e71-511a-3e98-949d-3f54561359cc (at 10.9.101.58@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2070f5c400, cur 1561504434 expire 1561504284 last 1561504207 Jun 25 16:13:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 16:14:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 16:14:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 16:18:16 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1dfeb79400, cur 1561504696 expire 1561504546 last 1561504469 Jun 25 16:18:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1af3fb9c00, cur 1561504715 expire 1561504565 last 1561504488 Jun 25 16:18:35 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 16:18:56 fir-md1-s1 kernel: Lustre: 21145:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f411f781800 x1634476328034336/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:1/0 lens 480/568 e 1 to 0 dl 1561504741 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 16:19:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e15f364b-b556-833b-9c7c-0e0e1407bf82 (at 10.9.0.62@o2ib4) reconnecting Jun 25 16:19:02 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jun 25 16:20:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jun 25 16:20:05 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jun 25 16:20:11 fir-md1-s1 kernel: LustreError: 23639:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561504721, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f3999702400/0x5d9ee6255bf0f49a lrc: 3/1,0 mode: --/PR res: [0x2c002a161:0xded:0x0].0x0 bits 0x40/0x0 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23639 timeout: 0 lvb_type: 0 Jun 25 16:21:10 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.9.102.21@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f1e8490f2c0/0x5d9ee6255bc9111b lrc: 3/0,0 mode: PW/PW res: [0x2c002a161:0xded:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.9.102.21@o2ib4 remote: 0xa578ba1cd90dd59f expref: 90 pid: 23573 timeout: 619930 lvb_type: 0 Jun 25 16:25:22 fir-md1-s1 kernel: Lustre: 97651:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1719849e00 x1635199131755344/t0(0) o101->018b4088-9100-7f5b-2709-38dd7f461ac7@10.8.8.29@o2ib6:27/0 lens 480/568 e 1 to 0 dl 1561505127 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 16:29:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 63c454e3-b29e-031b-b57d-b0e507f25d19 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2496e04800, cur 1561505383 expire 1561505233 last 1561505156 Jun 25 16:30:32 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ffa27290-6cf4-9b77-ab2a-7df1aa693fad (at 10.8.21.21@o2ib6) Jun 25 16:30:32 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jun 25 16:34:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 16:34:09 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 25 16:34:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) reconnecting Jun 25 16:34:12 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jun 25 16:34:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.68@o2ib6, removing former export from same NID Jun 25 16:37:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1da7775c00, cur 1561505876 expire 1561505726 last 1561505649 Jun 25 16:37:56 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jun 25 16:39:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 16:40:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jun 25 16:40:45 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jun 25 16:41:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 16:41:10 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 25 16:41:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.68@o2ib6, removing former export from same NID Jun 25 16:43:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.68@o2ib6, removing former export from same NID Jun 25 16:43:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.68@o2ib6, removing former export from same NID Jun 25 16:43:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 16:44:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) reconnecting Jun 25 16:44:18 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jun 25 16:54:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9cf6b6d2-1109-4702-108d-d26e95bd0151 (at 10.8.14.5@o2ib6) Jun 25 16:54:30 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jun 25 16:58:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1dfee81c00, cur 1561507093 expire 1561506943 last 1561506866 Jun 25 16:58:13 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 16:58:29 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f281d610800, cur 1561507109 expire 1561506959 last 1561506882 Jun 25 16:59:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 87da5719-38f8-e25f-27bd-899baebba0f4 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1e097a4400, cur 1561507194 expire 1561507044 last 1561506967 Jun 25 17:01:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 25 17:01:26 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 17:01:38 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561507291/real 1561507291] req@ffff8f1044215a00 x1636714449159776/t0(0) o104->fir-MDT0000@10.8.8.37@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561507298 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 25 17:01:45 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561507298/real 1561507298] req@ffff8f1044215a00 x1636714449159776/t0(0) o104->fir-MDT0000@10.8.8.37@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561507305 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 17:01:45 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 25 17:01:46 fir-md1-s1 kernel: Lustre: 23574:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f08486c2100 x1634476337723504/t0(0) o36->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:21/0 lens 512/2888 e 1 to 0 dl 1561507311 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 17:01:47 fir-md1-s1 kernel: Lustre: 10149:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3088a00300 x1634476337723712/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:22/0 lens 576/3264 e 1 to 0 dl 1561507312 ref 2 fl Interpret:/0/0 rc 0/0 Jun 25 17:02:00 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561507312/real 1561507312] req@ffff8f1044215a00 x1636714449159776/t0(0) o104->fir-MDT0000@10.8.8.37@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561507319 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 25 17:02:00 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jun 25 17:02:07 fir-md1-s1 kernel: LustreError: 23598:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.8.37@o2ib6) failed to reply to blocking AST (req@ffff8f1044215a00 x1636714449159776 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f1b1caea880/0x5d9ee62561f4661d lrc: 4/0,0 mode: CR/CR res: [0x2000297d4:0xab9b:0x0].0x0 bits 0x9/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.8.37@o2ib6 remote: 0xb50bab6d0e7b6fcf expref: 5430 pid: 21461 timeout: 622409 lvb_type: 0 Jun 25 17:02:07 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.8.37@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jun 25 17:02:07 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 36s: evicting client at 10.8.8.37@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1b1caea880/0x5d9ee62561f4661d lrc: 3/0,0 mode: CR/CR res: [0x2000297d4:0xab9b:0x0].0x0 bits 0x9/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.8.37@o2ib6 remote: 0xb50bab6d0e7b6fcf expref: 5431 pid: 21461 timeout: 0 lvb_type: 0 Jun 25 17:03:47 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34e1e7d000, cur 1561507427 expire 1561507277 last 1561507200 Jun 25 17:04:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 1e1769d3-ffba-a4ec-e5e5-cf0cf094a85d (at 10.8.8.37@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f250bd87400, cur 1561507454 expire 1561507304 last 1561507227 Jun 25 17:04:14 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 17:05:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f140f57a000, cur 1561507513 expire 1561507363 last 1561507286 Jun 25 17:05:13 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 17:12:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 25 17:12:04 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jun 25 17:12:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 17:12:04 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 25 17:12:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.0.64@o2ib4, removing former export from same NID Jun 25 17:14:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) reconnecting Jun 25 17:14:18 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 17:15:51 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f22d701d000, cur 1561508151 expire 1561508001 last 1561507924 Jun 25 17:16:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1a6b241c00, cur 1561508214 expire 1561508064 last 1561507987 Jun 25 17:16:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 17:17:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 17:18:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9b20a7cb-a3fc-d0ca-5cea-5de703dce72f (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24a6eea800, cur 1561508285 expire 1561508135 last 1561508058 Jun 25 17:19:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) in 168 seconds. I think it's dead, and I am evicting it. exp ffff8f24f3ea7c00, cur 1561508361 expire 1561508211 last 1561508193 Jun 25 17:19:21 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 17:20:20 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148ebd7400, cur 1561508420 expire 1561508270 last 1561508193 Jun 25 17:23:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 17:23:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.0.64@o2ib4, removing former export from same NID Jun 25 17:23:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 25 17:23:58 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jun 25 17:24:14 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25388da800, cur 1561508654 expire 1561508504 last 1561508427 Jun 25 17:24:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 25 17:24:19 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 17:25:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.64@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 17:38:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jun 25 17:38:13 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jun 25 17:42:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f20dbfc4000, cur 1561509720 expire 1561509570 last 1561509493 Jun 25 17:42:00 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 17:42:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.66@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 17:42:26 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 25 17:46:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f2c762800, cur 1561509973 expire 1561509823 last 1561509746 Jun 25 17:46:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1ef3be2c00, cur 1561510016 expire 1561509866 last 1561509789 Jun 25 17:46:56 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 17:48:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jun 25 17:48:21 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 25 17:52:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2521bef800, cur 1561510328 expire 1561510178 last 1561510101 Jun 25 17:53:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.66@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 17:53:23 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 25 17:53:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.0.66@o2ib6, removing former export from same NID Jun 25 17:53:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) reconnecting Jun 25 17:53:43 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 18:09:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) reconnecting Jun 25 18:09:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 18:09:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jun 25 18:09:42 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jun 25 18:11:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1a27aec800, cur 1561511492 expire 1561511342 last 1561511265 Jun 25 18:13:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1af42b1c00, cur 1561511609 expire 1561511459 last 1561511382 Jun 25 18:34:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jun 25 18:38:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 6d0f4c77-c27b-6d80-d629-873de917b74e (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2522978800, cur 1561513089 expire 1561512939 last 1561512862 Jun 25 18:38:09 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 18:38:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 111ade33-4633-d4f3-7359-6217f5551ac0 (at 10.8.14.9@o2ib6) Jun 25 18:39:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 8ea16e6d-a041-cebf-bc4c-b2c20885e699 (at 10.8.14.9@o2ib6) in 188 seconds. I think it's dead, and I am evicting it. exp ffff8f24ee4ae400, cur 1561513165 expire 1561513015 last 1561512977 Jun 25 18:40:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 8ea16e6d-a041-cebf-bc4c-b2c20885e699 (at 10.8.14.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2502d7bc00, cur 1561513206 expire 1561513056 last 1561512979 Jun 25 20:42:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) reconnecting Jun 25 20:42:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 25 20:42:07 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 20:42:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) reconnecting Jun 25 20:43:42 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1505447400, cur 1561520622 expire 1561520472 last 1561520395 Jun 25 20:43:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 20:45:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bd073587-8042-ffd0-09f1-ff79e8722875 (at 10.9.0.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2530f9e000, cur 1561520754 expire 1561520604 last 1561520527 Jun 25 20:47:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 25 20:47:17 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 20:47:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.0.63@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 25 20:51:04 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0846a2a000, cur 1561521064 expire 1561520914 last 1561520837 Jun 25 20:51:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 25 21:02:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jun 25 21:51:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to acd26ab4-a020-fbc0-1a40-f0e7d759131f (at 10.8.23.14@o2ib6) Jun 25 21:51:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 21:51:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6e7eede8-baef-e511-db4f-923a79b34ba3 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f451de70800, cur 1561524697 expire 1561524547 last 1561524470 Jun 25 22:29:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client c88d882f-e4f4-4b30-616b-f60f68016c23 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f19e686bc00, cur 1561526996 expire 1561526846 last 1561526769 Jun 25 22:29:56 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 25 22:30:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to acd26ab4-a020-fbc0-1a40-f0e7d759131f (at 10.8.23.14@o2ib6) Jun 25 22:30:03 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 00:00:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 939c0635-d3e5-7945-6eca-6a92a2676304 (at 10.9.101.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4535c8ec00, cur 1561532402 expire 1561532252 last 1561532175 Jun 26 00:00:02 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 00:00:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 939c0635-d3e5-7945-6eca-6a92a2676304 (at 10.9.101.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1506081400, cur 1561532415 expire 1561532265 last 1561532188 Jun 26 00:00:15 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 26 00:01:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client f017f489-eef8-cd54-4b70-e8f0166c7c7c (at 10.8.8.25@o2ib6) in 188 seconds. I think it's dead, and I am evicting it. exp ffff8f2522959000, cur 1561532478 expire 1561532328 last 1561532290 Jun 26 00:01:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client f017f489-eef8-cd54-4b70-e8f0166c7c7c (at 10.8.8.25@o2ib6) in 201 seconds. I think it's dead, and I am evicting it. exp ffff8f3509f55400, cur 1561532491 expire 1561532341 last 1561532290 Jun 26 00:01:57 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 36129863-f97a-d76f-0f90-11f02517721a (at 10.8.8.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1489ff5c00, cur 1561532517 expire 1561532367 last 1561532290 Jun 26 00:02:39 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jun 26 00:02:39 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 01:37:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 967359b8-6075-fa10-8749-55133a475ab0 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2520d55000, cur 1561538230 expire 1561538080 last 1561538003 Jun 26 01:54:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f15da91-4546-507e-8c99-9e08b5e219a4 (at 10.8.15.10@o2ib6) Jun 26 01:54:29 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 01:55:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 09fe1fc8-d186-6314-b715-72bcbbf4dcb1 (at 10.8.1.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2509c65800, cur 1561539336 expire 1561539186 last 1561539109 Jun 26 01:55:36 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 01:59:18 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b7a06525-6fdb-7245-d004-135045c5b952 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3ab6aa5800, cur 1561539558 expire 1561539408 last 1561539331 Jun 26 01:59:18 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 01:59:20 fir-md1-s1 kernel: LustreError: 97661:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.8@o2ib6) returned error from blocking AST (req@ffff8f1bdcfdc200 x1636715389943856 status -107 rc -107), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f24557dc5c0/0x5d9ee6262033f223 lrc: 4/0,0 mode: PR/PR res: [0x20002993d:0x1b0:0x0].0x0 bits 0x1b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.9.8@o2ib6 remote: 0x76a5494a455f2a91 expref: 31 pid: 50446 timeout: 654769 lvb_type: 0 Jun 26 01:59:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 4d708e92-2967-fb68-a999-b8fb560068d3 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1a682a9c00, cur 1561539560 expire 1561539410 last 1561539333 Jun 26 01:59:20 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 26 01:59:20 fir-md1-s1 kernel: LustreError: 97661:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Jun 26 01:59:20 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.9.8@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Jun 26 01:59:20 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 26 01:59:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ec76f1db-9c9b-bbe0-847f-90a9d517c8dc (at 10.8.9.8@o2ib6) Jun 26 01:59:31 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 02:12:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to aca88a5d-734b-f4a5-55fa-0e35d21bcb4e (at 10.8.0.65@o2ib6) Jun 26 02:12:06 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 02:24:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b4a2e41f-34ef-236e-f48b-7a4e4b82c56e (at 10.9.101.4@o2ib4) Jun 26 02:24:13 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 02:25:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.8.25@o2ib6) Jun 26 02:25:57 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 02:29:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to be77157a-c39a-b0a3-f5b0-4e7917893782 (at 10.8.1.35@o2ib6) Jun 26 02:29:00 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 26 04:01:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f15da91-4546-507e-8c99-9e08b5e219a4 (at 10.8.15.10@o2ib6) Jun 26 04:01:52 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 06:47:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f15da91-4546-507e-8c99-9e08b5e219a4 (at 10.8.15.10@o2ib6) Jun 26 06:47:19 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 11:44:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 0c122ba5-e660-84b0-99ae-db1f65f35f74 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1e17ea9000, cur 1561574671 expire 1561574521 last 1561574444 Jun 26 11:44:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ec76f1db-9c9b-bbe0-847f-90a9d517c8dc (at 10.8.9.8@o2ib6) Jun 26 11:44:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 11:44:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 0c122ba5-e660-84b0-99ae-db1f65f35f74 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0da9ed6400, cur 1561574681 expire 1561574531 last 1561574454 Jun 26 11:44:41 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 26 13:01:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 26320709-561f-90ed-6684-fea46854b319 (at 10.8.1.29@o2ib6) Jun 26 13:01:45 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 13:25:20 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 26 13:25:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3cd15e44-adf1-e977-3310-908c278e7f22 (at 10.8.0.68@o2ib6) reconnecting Jun 26 13:25:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jun 26 13:25:27 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 26 13:25:28 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 26 13:25:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3cd15e44-adf1-e977-3310-908c278e7f22 (at 10.8.0.68@o2ib6) reconnecting Jun 26 13:25:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jun 26 13:25:41 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 26 13:25:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3cd15e44-adf1-e977-3310-908c278e7f22 (at 10.8.0.68@o2ib6) reconnecting Jun 26 13:25:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jun 26 13:25:49 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 26 13:25:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3cd15e44-adf1-e977-3310-908c278e7f22 (at 10.8.0.68@o2ib6) reconnecting Jun 26 13:25:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jun 26 13:25:58 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 26 13:26:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3cd15e44-adf1-e977-3310-908c278e7f22 (at 10.8.0.68@o2ib6) reconnecting Jun 26 13:26:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jun 26 13:26:13 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 26 13:26:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3cd15e44-adf1-e977-3310-908c278e7f22 (at 10.8.0.68@o2ib6) reconnecting Jun 26 13:26:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jun 26 18:51:16 fir-md1-s1 kernel: LNetError: 20183:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jun 26 18:51:16 fir-md1-s1 kernel: LNetError: 20183:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Jun 26 20:08:10 fir-md1-s1 kernel: Lustre: 10197:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:08:10 fir-md1-s1 kernel: Lustre: 10197:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 75 previous similar messages Jun 26 20:09:08 fir-md1-s1 kernel: Lustre: 23555:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:09:08 fir-md1-s1 kernel: Lustre: 23555:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages Jun 26 20:15:28 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:15:28 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages Jun 26 20:23:56 fir-md1-s1 kernel: Lustre: 23578:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:23:56 fir-md1-s1 kernel: Lustre: 23578:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 102 previous similar messages Jun 26 20:24:06 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:24:06 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 26 20:26:43 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:26:43 fir-md1-s1 kernel: Lustre: 21411:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 47 previous similar messages Jun 26 20:30:13 fir-md1-s1 kernel: Lustre: 23672:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:30:13 fir-md1-s1 kernel: Lustre: 23672:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1375 previous similar messages Jun 26 20:42:00 fir-md1-s1 kernel: Lustre: 23683:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:42:00 fir-md1-s1 kernel: Lustre: 23683:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages Jun 26 20:42:34 fir-md1-s1 kernel: Lustre: 23556:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:42:34 fir-md1-s1 kernel: Lustre: 23556:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages Jun 26 20:47:21 fir-md1-s1 kernel: Lustre: 23578:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:47:21 fir-md1-s1 kernel: Lustre: 23578:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 24 previous similar messages Jun 26 20:53:40 fir-md1-s1 kernel: Lustre: 23683:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:54:27 fir-md1-s1 kernel: Lustre: 10506:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:54:27 fir-md1-s1 kernel: Lustre: 10506:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 169 previous similar messages Jun 26 20:58:15 fir-md1-s1 kernel: Lustre: 23594:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 20:58:15 fir-md1-s1 kernel: Lustre: 23594:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jun 26 21:01:54 fir-md1-s1 kernel: Lustre: 23594:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 21:43:00 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 26 21:43:00 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 516 previous similar messages Jun 26 21:44:21 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 26 21:44:21 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 40 previous similar messages Jun 26 21:47:00 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 26 21:47:00 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 27 previous similar messages Jun 26 22:08:33 fir-md1-s1 kernel: Lustre: 10304:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 26 22:08:33 fir-md1-s1 kernel: Lustre: 10304:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 23 previous similar messages Jun 26 22:58:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 26320709-561f-90ed-6684-fea46854b319 (at 10.8.1.29@o2ib6) Jun 26 22:58:09 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 27 00:51:42 fir-md1-s1 kernel: LustreError: 21545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 27 00:51:42 fir-md1-s1 kernel: LustreError: 21545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 47 previous similar messages Jun 27 00:55:52 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 27 01:03:11 fir-md1-s1 kernel: LustreError: 22157:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 27 01:03:11 fir-md1-s1 kernel: LustreError: 22157:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 27 01:12:03 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 27 01:12:03 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 27 01:20:15 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 27 01:20:15 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 27 01:21:01 fir-md1-s1 kernel: Lustre: 23672:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 27 01:31:12 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 27 01:31:12 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jun 27 01:55:57 fir-md1-s1 kernel: Lustre: 23632:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 27 01:57:27 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 27 03:01:01 fir-md1-s1 kernel: Lustre: 20983:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jun 27 03:35:12 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 27 03:35:12 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 27 03:36:33 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 27 03:36:33 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 48 previous similar messages Jun 27 03:39:24 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 27 03:39:24 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jun 27 03:46:14 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 27 03:46:14 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 27 04:29:20 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jun 27 11:57:14 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 11:58:30 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 11:58:30 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 50 previous similar messages Jun 27 12:01:01 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 12:01:01 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 112 previous similar messages Jun 27 12:06:01 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 12:06:01 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 226 previous similar messages Jun 27 12:16:02 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 12:16:02 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 461 previous similar messages Jun 27 12:26:03 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 12:26:03 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 459 previous similar messages Jun 27 12:36:03 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 12:36:03 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 460 previous similar messages Jun 27 12:46:04 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 12:46:04 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 449 previous similar messages Jun 27 12:56:05 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 12:56:05 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 457 previous similar messages Jun 27 13:06:05 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 13:06:05 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 457 previous similar messages Jun 27 13:16:06 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 13:16:06 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 455 previous similar messages Jun 27 13:26:07 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 13:26:07 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 456 previous similar messages Jun 27 13:36:08 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 13:36:08 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 454 previous similar messages Jun 27 13:46:08 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 13:46:08 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 450 previous similar messages Jun 27 13:56:09 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 13:56:09 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 450 previous similar messages Jun 27 14:06:10 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 14:06:10 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 459 previous similar messages Jun 27 14:16:10 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 14:16:10 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 458 previous similar messages Jun 27 14:26:12 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 14:26:12 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 452 previous similar messages Jun 27 14:36:12 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 14:36:12 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 456 previous similar messages Jun 27 14:46:13 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 14:46:13 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 452 previous similar messages Jun 27 14:56:13 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 14:56:13 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 451 previous similar messages Jun 27 15:06:13 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 15:06:13 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 447 previous similar messages Jun 27 15:16:13 fir-md1-s1 kernel: LustreError: 46549:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 15:16:13 fir-md1-s1 kernel: LustreError: 46549:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 451 previous similar messages Jun 27 15:26:14 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 15:26:14 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 448 previous similar messages Jun 27 15:36:15 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 15:36:15 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 450 previous similar messages Jun 27 15:43:50 fir-md1-s1 kernel: Lustre: 10589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561675423/real 1561675423] req@ffff8f10f8661200 x1636716613077104/t0(0) o106->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561675430 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 27 15:43:50 fir-md1-s1 kernel: Lustre: 10589:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jun 27 15:43:57 fir-md1-s1 kernel: Lustre: 23701:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561675430/real 1561675430] req@ffff8f0ae2ea7b00 x1636716613077152/t0(0) o106->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561675437 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 27 15:43:57 fir-md1-s1 kernel: Lustre: 23701:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 27 15:43:58 fir-md1-s1 kernel: Lustre: 20571:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f140aa09500 x1637002162748912/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:3/0 lens 480/568 e 1 to 0 dl 1561675443 ref 2 fl Interpret:/0/0 rc 0/0 Jun 27 15:43:58 fir-md1-s1 kernel: Lustre: 20571:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jun 27 15:44:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jun 27 15:44:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 27 15:44:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 27 15:44:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 27 15:44:04 fir-md1-s1 kernel: Lustre: 21410:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561675437/real 1561675437] req@ffff8f1cbd3cf500 x1636716613077120/t0(0) o106->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561675444 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 27 15:44:04 fir-md1-s1 kernel: Lustre: 21410:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 27 15:44:18 fir-md1-s1 kernel: Lustre: 21410:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561675451/real 1561675451] req@ffff8f1cbd3cf500 x1636716613077120/t0(0) o106->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561675458 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 27 15:44:18 fir-md1-s1 kernel: Lustre: 21410:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jun 27 15:44:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jun 27 15:44:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 27 15:44:39 fir-md1-s1 kernel: Lustre: 10589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561675472/real 1561675472] req@ffff8f10f8661200 x1636716613077104/t0(0) o106->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561675479 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 27 15:44:39 fir-md1-s1 kernel: Lustre: 10589:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jun 27 15:44:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jun 27 15:44:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 27 15:45:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jun 27 15:45:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 27 15:45:14 fir-md1-s1 kernel: Lustre: 23701:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561675507/real 1561675507] req@ffff8f0ae2ea7b00 x1636716613077152/t0(0) o106->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561675514 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 27 15:45:14 fir-md1-s1 kernel: Lustre: 23701:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jun 27 15:45:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jun 27 15:45:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 27 15:45:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 27 15:46:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jun 27 15:46:10 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 27 15:46:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jun 27 15:46:15 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 15:46:15 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 428 previous similar messages Jun 27 15:46:24 fir-md1-s1 kernel: Lustre: 23701:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561675577/real 1561675577] req@ffff8f0ae2ea7b00 x1636716613077152/t0(0) o106->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561675584 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jun 27 15:46:24 fir-md1-s1 kernel: Lustre: 23701:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Jun 27 15:46:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client b09d4c25-b109-b30c-132e-6a644105be34 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34ea696400, cur 1561675597 expire 1561675447 last 1561675370 Jun 27 15:46:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client b09d4c25-b109-b30c-132e-6a644105be34 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1fa790b000, cur 1561675612 expire 1561675462 last 1561675385 Jun 27 15:46:52 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 27 15:46:52 fir-md1-s1 kernel: Lustre: 21410:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (188:1s); client may timeout. req@ffff8f0cc4698900 x1637002162748944/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:3/0 lens 480/536 e 1 to 0 dl 1561675611 ref 1 fl Complete:/0/0 rc 301/301 Jun 27 15:46:52 fir-md1-s1 kernel: Lustre: 21410:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1453 previous similar messages Jun 27 15:48:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 27 15:48:21 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 27 15:50:54 fir-md1-s1 kernel: LustreError: 21460:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f2d39ed9500 x1636716686208928/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 27 15:50:54 fir-md1-s1 kernel: LustreError: 21460:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Jun 27 15:50:56 fir-md1-s1 kernel: LustreError: 21446:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f2d1fe59500 x1636716686210688/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 27 15:50:56 fir-md1-s1 kernel: LustreError: 21446:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jun 27 15:51:09 fir-md1-s1 kernel: Lustre: 97641:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f19ad73a100 x1636512232467120/t0(0) o101->3429bec6-fe2a-19ec-4f0c-bb576fed4ff4@10.8.29.4@o2ib6:14/0 lens 480/568 e 1 to 0 dl 1561675874 ref 2 fl Interpret:/0/0 rc 0/0 Jun 27 15:51:09 fir-md1-s1 kernel: Lustre: 97641:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jun 27 15:51:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3429bec6-fe2a-19ec-4f0c-bb576fed4ff4 (at 10.8.29.4@o2ib6) reconnecting Jun 27 15:51:15 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 27 15:51:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0ed884ea-fa51-544e-85e4-1d3a8c288fe4 (at 10.8.29.4@o2ib6) Jun 27 15:51:15 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 27 15:51:35 fir-md1-s1 kernel: LustreError: 97644:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f169db1ef00 x1636716686243552/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 27 15:51:35 fir-md1-s1 kernel: LustreError: 97644:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Jun 27 15:52:24 fir-md1-s1 kernel: LustreError: 97670:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561675854, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1ab63c4a40/0x5d9ee62884870864 lrc: 3/0,1 mode: --/PW res: [0x2000222aa:0x10f:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97670 timeout: 0 lvb_type: 0 Jun 27 15:52:24 fir-md1-s1 kernel: LustreError: 97670:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jun 27 15:52:36 fir-md1-s1 kernel: Lustre: 23578:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f0c54094b00 x1637002172029856/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:11/0 lens 592/3264 e 0 to 0 dl 1561675961 ref 2 fl Interpret:/0/0 rc 0/0 Jun 27 15:52:36 fir-md1-s1 kernel: Lustre: 23578:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jun 27 15:53:05 fir-md1-s1 kernel: LustreError: 97644:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561675895, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f22e074ec00/0x5d9ee62884a9d9f6 lrc: 3/0,1 mode: --/EX res: [0x200029c2e:0x62:0x0].0x0 bits 0x21/0x0 rrc: 5 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 97644 timeout: 0 lvb_type: 0 Jun 27 15:53:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID Jun 27 15:53:23 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f340ef8f980/0x5d9ee6287f84cd5d lrc: 3/0,0 mode: PR/PR res: [0x2000222aa:0x10f:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0xe063eaa9a5d7f7c1 expref: 90843 pid: 97666 timeout: 791063 lvb_type: 0 Jun 27 15:53:39 fir-md1-s1 kernel: LustreError: 97644:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1d48b63c00 x1636716686375312/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 27 15:53:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3429bec6-fe2a-19ec-4f0c-bb576fed4ff4 (at 10.8.29.4@o2ib6) reconnecting Jun 27 15:53:42 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jun 27 15:53:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0ed884ea-fa51-544e-85e4-1d3a8c288fe4 (at 10.8.29.4@o2ib6) Jun 27 15:53:42 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jun 27 15:54:14 fir-md1-s1 kernel: LNet: Service thread pid 97670 was inactive for 200.24s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 27 15:54:14 fir-md1-s1 kernel: Pid: 97670, comm: mdt01_109 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 27 15:54:14 fir-md1-s1 kernel: Call Trace: Jun 27 15:54:14 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 27 15:54:14 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 27 15:54:14 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jun 27 15:54:14 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 27 15:54:14 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Jun 27 15:54:14 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Jun 27 15:54:14 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Jun 27 15:54:14 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jun 27 15:54:14 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jun 27 15:54:14 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jun 27 15:54:14 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 27 15:54:14 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 27 15:54:14 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 27 15:54:14 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 27 15:54:14 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 27 15:54:14 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 27 15:54:14 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 27 15:54:14 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561676054.97670 Jun 27 15:54:22 fir-md1-s1 kernel: LNet: Service thread pid 97670 completed after 208.25s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 27 15:56:16 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 15:56:16 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 448 previous similar messages Jun 27 16:06:16 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 16:06:16 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 27 16:16:18 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 16:16:18 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 464 previous similar messages Jun 27 16:26:19 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 16:26:19 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 467 previous similar messages Jun 27 16:36:21 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 16:36:21 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 464 previous similar messages Jun 27 16:46:21 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 16:46:21 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 465 previous similar messages Jun 27 16:56:21 fir-md1-s1 kernel: LustreError: 46578:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 16:56:21 fir-md1-s1 kernel: LustreError: 46578:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 27 17:06:22 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 17:06:22 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 466 previous similar messages Jun 27 17:16:23 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 17:16:23 fir-md1-s1 kernel: LustreError: 22648:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 464 previous similar messages Jun 27 17:26:23 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 17:26:23 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 459 previous similar messages Jun 27 17:36:24 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 17:36:24 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 462 previous similar messages Jun 27 17:46:26 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 17:46:26 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 454 previous similar messages Jun 27 17:56:26 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 17:56:26 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 455 previous similar messages Jun 27 18:06:27 fir-md1-s1 kernel: LustreError: 46529:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 18:06:27 fir-md1-s1 kernel: LustreError: 46529:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 431 previous similar messages Jun 27 18:16:27 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 18:16:27 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 404 previous similar messages Jun 27 18:26:28 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 18:26:28 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 397 previous similar messages Jun 27 18:36:29 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 18:36:29 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 396 previous similar messages Jun 27 18:46:29 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 18:46:29 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 395 previous similar messages Jun 27 18:48:54 fir-md1-s1 kernel: Lustre: 23565:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jun 27 18:56:29 fir-md1-s1 kernel: LustreError: 46561:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 18:56:29 fir-md1-s1 kernel: LustreError: 46561:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 394 previous similar messages Jun 27 19:06:31 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 19:06:31 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 395 previous similar messages Jun 27 19:16:32 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 19:16:32 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 393 previous similar messages Jun 27 19:26:32 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 19:26:32 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 430 previous similar messages Jun 27 19:36:33 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 19:36:33 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 433 previous similar messages Jun 27 19:46:33 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 19:46:33 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 408 previous similar messages Jun 27 19:56:34 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 19:56:34 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 411 previous similar messages Jun 27 20:06:35 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 20:06:35 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 454 previous similar messages Jun 27 20:16:36 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 20:16:36 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 456 previous similar messages Jun 27 20:26:37 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 20:26:37 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 453 previous similar messages Jun 27 20:36:38 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 20:36:38 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 457 previous similar messages Jun 27 20:46:38 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 20:46:38 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 461 previous similar messages Jun 27 20:56:38 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 20:56:38 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 455 previous similar messages Jun 27 21:06:39 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 21:06:39 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 458 previous similar messages Jun 27 21:16:39 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 21:16:39 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 454 previous similar messages Jun 27 21:26:40 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 21:26:40 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 453 previous similar messages Jun 27 21:36:41 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 21:36:41 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 459 previous similar messages Jun 27 21:46:42 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 21:46:42 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 455 previous similar messages Jun 27 21:56:43 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 21:56:43 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 453 previous similar messages Jun 27 22:06:44 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 22:06:44 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 461 previous similar messages Jun 27 22:16:44 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 22:16:44 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 27 22:26:46 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 22:26:46 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 467 previous similar messages Jun 27 22:36:47 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 22:36:47 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 444 previous similar messages Jun 27 22:46:48 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 22:46:48 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 452 previous similar messages Jun 27 22:56:48 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 22:56:48 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 489 previous similar messages Jun 27 23:06:49 fir-md1-s1 kernel: LustreError: 57787:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 23:06:49 fir-md1-s1 kernel: LustreError: 57787:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 498 previous similar messages Jun 27 23:16:50 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 23:16:50 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 495 previous similar messages Jun 27 23:26:50 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 23:26:50 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 499 previous similar messages Jun 27 23:36:51 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 23:36:51 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 509 previous similar messages Jun 27 23:46:52 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 23:46:52 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 502 previous similar messages Jun 27 23:56:52 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 27 23:56:52 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 507 previous similar messages Jun 28 00:06:53 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 00:06:53 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 466 previous similar messages Jun 28 00:16:54 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 00:16:54 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 474 previous similar messages Jun 28 00:26:55 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 00:26:55 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 479 previous similar messages Jun 28 00:36:55 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 00:36:55 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 509 previous similar messages Jun 28 00:46:55 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 00:46:55 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 533 previous similar messages Jun 28 00:56:55 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 00:56:55 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 527 previous similar messages Jun 28 01:06:56 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 01:06:56 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 524 previous similar messages Jun 28 01:16:57 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 01:16:57 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 529 previous similar messages Jun 28 01:26:58 fir-md1-s1 kernel: LustreError: 46561:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 01:26:58 fir-md1-s1 kernel: LustreError: 46561:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 549 previous similar messages Jun 28 01:36:58 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 01:36:58 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 543 previous similar messages Jun 28 01:46:58 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 01:46:58 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 541 previous similar messages Jun 28 01:56:58 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 01:56:58 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 540 previous similar messages Jun 28 02:07:00 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 02:07:00 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 28 02:17:01 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 02:17:01 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 401 previous similar messages Jun 28 02:27:02 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 02:27:02 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 398 previous similar messages Jun 28 02:37:03 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 02:37:03 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 401 previous similar messages Jun 28 02:47:03 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 02:47:03 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jun 28 02:57:04 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 02:57:04 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 398 previous similar messages Jun 28 03:07:04 fir-md1-s1 kernel: LustreError: 57787:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 03:07:04 fir-md1-s1 kernel: LustreError: 57787:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jun 28 03:17:05 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 03:17:05 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 401 previous similar messages Jun 28 03:27:06 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 03:27:06 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jun 28 03:37:07 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 03:37:07 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 399 previous similar messages Jun 28 03:47:08 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 03:47:08 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 396 previous similar messages Jun 28 03:57:09 fir-md1-s1 kernel: LustreError: 46593:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 03:57:09 fir-md1-s1 kernel: LustreError: 46593:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 387 previous similar messages Jun 28 04:07:10 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 04:07:10 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 365 previous similar messages Jun 28 04:17:11 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 04:17:11 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 378 previous similar messages Jun 28 04:27:12 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 04:27:12 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 387 previous similar messages Jun 28 04:37:12 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 04:37:12 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 378 previous similar messages Jun 28 04:47:12 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 04:47:12 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 392 previous similar messages Jun 28 04:57:13 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 04:57:13 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jun 28 05:07:14 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 05:07:14 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 382 previous similar messages Jun 28 05:17:14 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 05:17:14 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 387 previous similar messages Jun 28 05:27:15 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 05:27:15 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 373 previous similar messages Jun 28 05:37:16 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 05:37:16 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 387 previous similar messages Jun 28 05:47:17 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 05:47:17 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 378 previous similar messages Jun 28 05:57:17 fir-md1-s1 kernel: LustreError: 21717:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 05:57:17 fir-md1-s1 kernel: LustreError: 21717:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 386 previous similar messages Jun 28 06:04:46 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561727079/real 1561727079] req@ffff8f10fc24dd00 x1636717505315616/t0(0) o106->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561727086 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 28 06:04:46 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jun 28 06:04:54 fir-md1-s1 kernel: Lustre: 10506:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0ad0e0ec00 x1637002685958656/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:29/0 lens 480/568 e 1 to 0 dl 1561727099 ref 2 fl Interpret:/0/0 rc 0/0 Jun 28 06:05:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 4c6d21f6-3e09-6b98-bf50-a29faf23fa85 (at 10.8.9.9@o2ib6) reconnecting Jun 28 06:05:00 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 28 06:05:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 28 06:05:00 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 28 06:05:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 4c6d21f6-3e09-6b98-bf50-a29faf23fa85 (at 10.8.9.9@o2ib6) reconnecting Jun 28 06:05:44 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 28 06:05:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 28 06:05:44 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 28 06:07:19 fir-md1-s1 kernel: LustreError: 21291:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 06:07:19 fir-md1-s1 kernel: LustreError: 21291:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 391 previous similar messages Jun 28 06:17:19 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 06:17:19 fir-md1-s1 kernel: LustreError: 25633:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 376 previous similar messages Jun 28 06:27:20 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 06:27:20 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 381 previous similar messages Jun 28 06:37:22 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 06:37:22 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 370 previous similar messages Jun 28 06:47:22 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 06:47:22 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 06:57:23 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 06:57:23 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 372 previous similar messages Jun 28 07:07:24 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 07:07:24 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 390 previous similar messages Jun 28 07:17:24 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 07:17:24 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jun 28 07:27:25 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 07:27:25 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 388 previous similar messages Jun 28 07:37:25 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 07:37:25 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 07:47:26 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 07:47:26 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 07:57:28 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 07:57:28 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 08:07:29 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 08:07:29 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 382 previous similar messages Jun 28 08:17:30 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 08:17:30 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 08:27:31 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 08:27:31 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 08:37:32 fir-md1-s1 kernel: LustreError: 46561:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 08:37:32 fir-md1-s1 kernel: LustreError: 46561:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 08:47:33 fir-md1-s1 kernel: LustreError: 22226:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 08:47:33 fir-md1-s1 kernel: LustreError: 22226:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 08:57:33 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 08:57:33 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 382 previous similar messages Jun 28 09:07:34 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 09:07:34 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 09:17:35 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 09:17:35 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 383 previous similar messages Jun 28 09:27:36 fir-md1-s1 kernel: LustreError: 57787:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 09:27:36 fir-md1-s1 kernel: LustreError: 57787:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 384 previous similar messages Jun 28 09:37:37 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 09:37:37 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 09:47:37 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 09:47:37 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 384 previous similar messages Jun 28 09:57:37 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 09:57:37 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 382 previous similar messages Jun 28 10:07:37 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 10:07:37 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 380 previous similar messages Jun 28 10:17:39 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 10:17:39 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 367 previous similar messages Jun 28 10:27:40 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 10:27:40 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 10:33:39 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jun 28 10:37:41 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 10:37:41 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jun 28 10:47:42 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 10:47:42 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 383 previous similar messages Jun 28 10:57:42 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 10:57:42 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 380 previous similar messages Jun 28 11:07:43 fir-md1-s1 kernel: LustreError: 22226:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 11:07:43 fir-md1-s1 kernel: LustreError: 22226:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 379 previous similar messages Jun 28 11:17:44 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 11:17:44 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 377 previous similar messages Jun 28 11:27:45 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 11:27:45 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 368 previous similar messages Jun 28 11:37:46 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 11:37:46 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 374 previous similar messages Jun 28 11:47:47 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 11:47:47 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 381 previous similar messages Jun 28 11:57:48 fir-md1-s1 kernel: LustreError: 57787:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 11:57:48 fir-md1-s1 kernel: LustreError: 57787:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 382 previous similar messages Jun 28 12:07:49 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 12:07:49 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 428 previous similar messages Jun 28 12:17:50 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 12:17:50 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 491 previous similar messages Jun 28 12:27:50 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 12:27:50 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 500 previous similar messages Jun 28 12:37:50 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 12:37:50 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 498 previous similar messages Jun 28 12:47:51 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 12:47:51 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 494 previous similar messages Jun 28 12:57:52 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 12:57:52 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 490 previous similar messages Jun 28 13:07:52 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 13:07:52 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 499 previous similar messages Jun 28 13:17:52 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 13:17:52 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 501 previous similar messages Jun 28 13:27:53 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 13:27:53 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 503 previous similar messages Jun 28 13:31:01 fir-md1-s1 kernel: Lustre: 23689:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jun 28 13:37:53 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 13:37:53 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 501 previous similar messages Jun 28 13:47:54 fir-md1-s1 kernel: LustreError: 22226:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 13:47:54 fir-md1-s1 kernel: LustreError: 22226:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 502 previous similar messages Jun 28 13:50:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 52b5dc52-8a4c-f64c-7b51-91709e30d8ba (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f326b3d1800, cur 1561755001 expire 1561754851 last 1561754774 Jun 28 13:50:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 52b5dc52-8a4c-f64c-7b51-91709e30d8ba (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f121c50b000, cur 1561755008 expire 1561754858 last 1561754781 Jun 28 13:52:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d8cc7b58-ee01-5501-ca65-c659f4724147 (at 10.9.106.54@o2ib4) Jun 28 13:57:55 fir-md1-s1 kernel: LustreError: 21713:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 13:57:55 fir-md1-s1 kernel: LustreError: 21713:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 499 previous similar messages Jun 28 14:07:56 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 14:07:56 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 487 previous similar messages Jun 28 14:17:56 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 14:17:56 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 503 previous similar messages Jun 28 14:27:57 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 14:27:57 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 500 previous similar messages Jun 28 14:37:57 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 14:37:57 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 502 previous similar messages Jun 28 14:47:58 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 14:47:58 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 502 previous similar messages Jun 28 14:57:59 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 14:57:59 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 504 previous similar messages Jun 28 15:07:59 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 15:07:59 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 504 previous similar messages Jun 28 15:18:00 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 15:18:00 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 499 previous similar messages Jun 28 15:28:00 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 15:28:00 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 506 previous similar messages Jun 28 15:38:00 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 15:38:00 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 499 previous similar messages Jun 28 15:48:00 fir-md1-s1 kernel: LustreError: 46575:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 15:48:00 fir-md1-s1 kernel: LustreError: 46575:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 472 previous similar messages Jun 28 15:58:01 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 15:58:01 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 527 previous similar messages Jun 28 16:08:01 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 16:08:01 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 516 previous similar messages Jun 28 16:13:22 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 28 16:13:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 172b52dd-0b9e-12f8-c21f-947aedff05a0 (at 10.8.18.2@o2ib6) reconnecting Jun 28 16:13:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 2727f5d4-463f-2044-b04c-92df44e40c7d (at 10.8.18.2@o2ib6) Jun 28 16:13:53 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 28 16:14:35 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 28 16:15:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 7ab2f51d-a689-9f2c-be74-3bf003bf5840 (at 10.8.0.66@o2ib6) reconnecting Jun 28 16:15:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jun 28 16:18:01 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 16:18:01 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 538 previous similar messages Jun 28 16:28:02 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 16:28:02 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 536 previous similar messages Jun 28 16:38:02 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 16:38:02 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 536 previous similar messages Jun 28 16:48:03 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 16:48:03 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 534 previous similar messages Jun 28 16:58:04 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 16:58:04 fir-md1-s1 kernel: LustreError: 46590:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 527 previous similar messages Jun 28 17:08:05 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 17:08:05 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 532 previous similar messages Jun 28 17:18:06 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 17:18:06 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 528 previous similar messages Jun 28 17:28:07 fir-md1-s1 kernel: LustreError: 46549:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 17:28:07 fir-md1-s1 kernel: LustreError: 46549:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 535 previous similar messages Jun 28 17:38:07 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 17:38:07 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 524 previous similar messages Jun 28 17:48:08 fir-md1-s1 kernel: LustreError: 46533:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 17:48:08 fir-md1-s1 kernel: LustreError: 46533:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 515 previous similar messages Jun 28 17:58:09 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 17:58:09 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 527 previous similar messages Jun 28 18:03:03 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 28 18:03:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6660433e-6178-3b9d-5600-564c37c5d5bd (at 10.8.8.26@o2ib6) reconnecting Jun 28 18:03:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 4bf93a7c-5f27-067e-124f-bc871b3eff21 (at 10.8.8.26@o2ib6) Jun 28 18:08:09 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 18:08:09 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 482 previous similar messages Jun 28 18:18:10 fir-md1-s1 kernel: LustreError: 21291:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 18:18:10 fir-md1-s1 kernel: LustreError: 21291:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 410 previous similar messages Jun 28 18:28:10 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 18:28:10 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 411 previous similar messages Jun 28 18:38:11 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 18:38:11 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 410 previous similar messages Jun 28 18:48:12 fir-md1-s1 kernel: LustreError: 46561:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 18:48:12 fir-md1-s1 kernel: LustreError: 46561:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 407 previous similar messages Jun 28 18:58:13 fir-md1-s1 kernel: LustreError: 46533:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 18:58:13 fir-md1-s1 kernel: LustreError: 46533:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 411 previous similar messages Jun 28 19:08:13 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 19:08:13 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 405 previous similar messages Jun 28 19:18:14 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 19:18:14 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 409 previous similar messages Jun 28 19:28:14 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 19:28:14 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 407 previous similar messages Jun 28 19:38:14 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 19:38:14 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 408 previous similar messages Jun 28 19:48:15 fir-md1-s1 kernel: LustreError: 22157:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 19:48:15 fir-md1-s1 kernel: LustreError: 22157:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 405 previous similar messages Jun 28 19:58:16 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 19:58:16 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 413 previous similar messages Jun 28 20:08:17 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 20:08:17 fir-md1-s1 kernel: LustreError: 21390:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 400 previous similar messages Jun 28 20:18:17 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 20:18:17 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 400 previous similar messages Jun 28 20:28:19 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 20:28:19 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 408 previous similar messages Jun 28 20:38:19 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 20:38:19 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 28 20:48:20 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 20:48:20 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 407 previous similar messages Jun 28 20:58:21 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 20:58:21 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 406 previous similar messages Jun 28 21:08:21 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 21:08:21 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 394 previous similar messages Jun 28 21:18:22 fir-md1-s1 kernel: LustreError: 46575:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 21:18:22 fir-md1-s1 kernel: LustreError: 46575:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 368 previous similar messages Jun 28 21:28:24 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 21:28:24 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 401 previous similar messages Jun 28 21:38:24 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 21:38:24 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 409 previous similar messages Jun 28 21:48:24 fir-md1-s1 kernel: LustreError: 46575:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 21:48:24 fir-md1-s1 kernel: LustreError: 46575:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 438 previous similar messages Jun 28 21:58:24 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 21:58:24 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 454 previous similar messages Jun 28 22:08:24 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 22:08:24 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 476 previous similar messages Jun 28 22:18:25 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 22:18:25 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 490 previous similar messages Jun 28 22:28:26 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 22:28:26 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 507 previous similar messages Jun 28 22:38:27 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 22:38:27 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 522 previous similar messages Jun 28 22:48:27 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 22:48:27 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 515 previous similar messages Jun 28 22:53:52 fir-md1-s1 kernel: Lustre: 23650:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jun 28 22:58:27 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 22:58:27 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 516 previous similar messages Jun 28 23:08:29 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 23:08:29 fir-md1-s1 kernel: LustreError: 21542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 513 previous similar messages Jun 28 23:18:29 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 23:18:29 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 517 previous similar messages Jun 28 23:28:29 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 23:28:29 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 519 previous similar messages Jun 28 23:38:30 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 23:38:30 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 479 previous similar messages Jun 28 23:48:30 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 23:48:30 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 28 23:58:32 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 28 23:58:32 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 470 previous similar messages Jun 29 00:08:33 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 00:08:33 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 472 previous similar messages Jun 29 00:18:33 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 00:18:33 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 473 previous similar messages Jun 29 00:28:34 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 00:28:34 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 472 previous similar messages Jun 29 00:38:34 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 00:38:34 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 473 previous similar messages Jun 29 00:48:35 fir-md1-s1 kernel: LustreError: 46549:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 00:48:35 fir-md1-s1 kernel: LustreError: 46549:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 479 previous similar messages Jun 29 00:58:36 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 00:58:36 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 476 previous similar messages Jun 29 01:08:36 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 01:08:36 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 477 previous similar messages Jun 29 01:18:37 fir-md1-s1 kernel: LustreError: 46533:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 01:18:37 fir-md1-s1 kernel: LustreError: 46533:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 469 previous similar messages Jun 29 01:28:37 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 01:28:37 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 482 previous similar messages Jun 29 01:38:38 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 01:38:38 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 476 previous similar messages Jun 29 01:48:39 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 01:48:39 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 479 previous similar messages Jun 29 01:58:40 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 01:58:40 fir-md1-s1 kernel: LustreError: 25997:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 481 previous similar messages Jun 29 02:08:42 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 02:08:42 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 483 previous similar messages Jun 29 02:18:42 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 02:18:42 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 485 previous similar messages Jun 29 02:28:43 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 02:28:43 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 483 previous similar messages Jun 29 02:38:44 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 02:38:44 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 482 previous similar messages Jun 29 02:48:44 fir-md1-s1 kernel: LustreError: 25634:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 02:48:44 fir-md1-s1 kernel: LustreError: 25634:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 481 previous similar messages Jun 29 02:58:45 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 02:58:45 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 479 previous similar messages Jun 29 03:08:46 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 03:08:46 fir-md1-s1 kernel: LustreError: 21712:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 482 previous similar messages Jun 29 03:18:46 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 03:18:46 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 478 previous similar messages Jun 29 03:28:47 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 03:28:47 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 479 previous similar messages Jun 29 03:38:48 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 03:38:48 fir-md1-s1 kernel: LustreError: 22156:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 427 previous similar messages Jun 29 03:42:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9d4f346d-e38b-6c6e-266e-3da4c47c24e4 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24eff97c00, cur 1561804943 expire 1561804793 last 1561804716 Jun 29 03:42:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 29 03:43:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c6748fa-faf9-dbf4-7576-e7e488da698d (at 10.8.11.9@o2ib6) Jun 29 03:48:48 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 03:48:48 fir-md1-s1 kernel: LustreError: 22427:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 455 previous similar messages Jun 29 03:58:50 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 03:58:50 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 465 previous similar messages Jun 29 04:08:50 fir-md1-s1 kernel: LustreError: 21545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 04:08:50 fir-md1-s1 kernel: LustreError: 21545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 464 previous similar messages Jun 29 04:15:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d5062c59-a286-2049-232b-def850bbc374 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1a7ce4a000, cur 1561806941 expire 1561806791 last 1561806714 Jun 29 04:15:41 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 04:16:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c6748fa-faf9-dbf4-7576-e7e488da698d (at 10.8.11.9@o2ib6) Jun 29 04:16:59 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 04:18:50 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 04:18:50 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 472 previous similar messages Jun 29 04:28:51 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 04:28:51 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 465 previous similar messages Jun 29 04:38:52 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 04:38:52 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 466 previous similar messages Jun 29 04:48:52 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 04:48:52 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 460 previous similar messages Jun 29 04:56:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ae08e35a-e0d0-58b5-17ae-e4363256cb18 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1bdc869800, cur 1561809404 expire 1561809254 last 1561809177 Jun 29 04:56:44 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 04:57:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c6748fa-faf9-dbf4-7576-e7e488da698d (at 10.8.11.9@o2ib6) Jun 29 04:57:47 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 04:58:53 fir-md1-s1 kernel: LustreError: 46578:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 04:58:53 fir-md1-s1 kernel: LustreError: 46578:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 464 previous similar messages Jun 29 05:08:54 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 05:08:54 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 461 previous similar messages Jun 29 05:18:55 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 05:18:55 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 29 05:28:55 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 05:28:55 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 433 previous similar messages Jun 29 05:38:56 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 05:38:56 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 465 previous similar messages Jun 29 05:48:57 fir-md1-s1 kernel: LustreError: 25634:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 05:48:57 fir-md1-s1 kernel: LustreError: 25634:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 461 previous similar messages Jun 29 05:58:58 fir-md1-s1 kernel: LustreError: 46578:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 05:58:58 fir-md1-s1 kernel: LustreError: 46578:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 29 06:08:59 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 06:08:59 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 29 06:18:59 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 06:18:59 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 464 previous similar messages Jun 29 06:28:59 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 06:28:59 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 459 previous similar messages Jun 29 06:39:00 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 06:39:00 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 453 previous similar messages Jun 29 06:49:01 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 06:49:01 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 459 previous similar messages Jun 29 06:59:01 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 06:59:01 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 469 previous similar messages Jun 29 07:09:02 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 07:09:02 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 29 07:19:02 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 07:19:02 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 29 07:29:02 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 07:29:02 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 29 07:39:03 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 07:39:03 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 466 previous similar messages Jun 29 07:49:04 fir-md1-s1 kernel: LustreError: 25972:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 07:49:04 fir-md1-s1 kernel: LustreError: 25972:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 465 previous similar messages Jun 29 07:59:05 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 07:59:05 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 465 previous similar messages Jun 29 08:09:05 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 08:09:05 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 440 previous similar messages Jun 29 08:19:06 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 08:19:06 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 458 previous similar messages Jun 29 08:29:06 fir-md1-s1 kernel: LustreError: 21713:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 08:29:06 fir-md1-s1 kernel: LustreError: 21713:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 469 previous similar messages Jun 29 08:39:07 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 08:39:07 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 462 previous similar messages Jun 29 08:46:32 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jun 29 08:46:32 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 08:46:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 82c7213a-dc0a-1b63-00e6-606d680853e3 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3a06e57c00, cur 1561823196 expire 1561823046 last 1561822969 Jun 29 08:46:36 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 08:49:08 fir-md1-s1 kernel: LustreError: 46552:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 08:49:08 fir-md1-s1 kernel: LustreError: 46552:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 458 previous similar messages Jun 29 08:59:09 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 08:59:09 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 519 previous similar messages Jun 29 09:09:10 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 09:09:10 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 543 previous similar messages Jun 29 09:19:10 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 29 09:19:10 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 564 previous similar messages Jun 29 09:29:10 fir-md1-s1 kernel: LustreError: 46593:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 09:29:10 fir-md1-s1 kernel: LustreError: 46593:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 601 previous similar messages Jun 29 09:39:10 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 09:39:10 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 696 previous similar messages Jun 29 09:49:12 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 09:49:12 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 665 previous similar messages Jun 29 09:59:12 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 09:59:12 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 651 previous similar messages Jun 29 10:09:13 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 10:09:13 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 646 previous similar messages Jun 29 10:14:58 fir-md1-s1 kernel: sched: RT throttling activated Jun 29 10:19:14 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 10:19:14 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 600 previous similar messages Jun 29 10:29:14 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 10:29:14 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 546 previous similar messages Jun 29 10:39:14 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 10:39:14 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 588 previous similar messages Jun 29 10:49:16 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 10:49:16 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 555 previous similar messages Jun 29 10:59:17 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 10:59:17 fir-md1-s1 kernel: LustreError: 21540:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 545 previous similar messages Jun 29 11:09:17 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 11:09:17 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 493 previous similar messages Jun 29 11:19:17 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 11:19:17 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 462 previous similar messages Jun 29 11:29:18 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 11:29:18 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 460 previous similar messages Jun 29 11:39:19 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 11:39:19 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 442 previous similar messages Jun 29 11:49:19 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 11:49:19 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 445 previous similar messages Jun 29 11:59:20 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 11:59:20 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 448 previous similar messages Jun 29 12:09:20 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 12:09:20 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 29 12:19:21 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 12:19:21 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 465 previous similar messages Jun 29 12:29:22 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 12:29:22 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 457 previous similar messages Jun 29 12:39:23 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 12:39:23 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 470 previous similar messages Jun 29 12:49:24 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 12:49:24 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 474 previous similar messages Jun 29 12:59:24 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 12:59:24 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 29 13:09:24 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 13:09:24 fir-md1-s1 kernel: LustreError: 21448:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 472 previous similar messages Jun 29 13:19:25 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 13:19:25 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 29 13:29:25 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 13:29:25 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 29 13:39:25 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 13:39:25 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 485 previous similar messages Jun 29 13:49:26 fir-md1-s1 kernel: LustreError: 46549:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 13:49:26 fir-md1-s1 kernel: LustreError: 46549:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 494 previous similar messages Jun 29 13:51:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9991ec5a-a329-b00c-1b36-c9ef203c13d2 (at 10.8.1.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f250077fc00, cur 1561841469 expire 1561841319 last 1561841242 Jun 29 13:51:09 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 13:51:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f15da91-4546-507e-8c99-9e08b5e219a4 (at 10.8.15.10@o2ib6) Jun 29 13:51:57 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 13:52:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d5a8de86-e2c0-2c49-971c-021289d53cbe (at 10.8.1.31@o2ib6) Jun 29 13:52:45 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 13:59:28 fir-md1-s1 kernel: LustreError: 46533:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 13:59:28 fir-md1-s1 kernel: LustreError: 46533:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 431 previous similar messages Jun 29 14:09:28 fir-md1-s1 kernel: LustreError: 46578:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 14:09:28 fir-md1-s1 kernel: LustreError: 46578:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 29 14:19:28 fir-md1-s1 kernel: LustreError: 21565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 14:19:28 fir-md1-s1 kernel: LustreError: 21565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 401 previous similar messages Jun 29 14:29:29 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 14:29:29 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 29 14:39:29 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 14:39:29 fir-md1-s1 kernel: LustreError: 27603:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 404 previous similar messages Jun 29 14:43:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 3a81a303-95c2-a3aa-25be-f3ca1eccf64d (at 10.9.104.31@o2ib4) Jun 29 14:43:31 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 14:44:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 18cb183b-1663-4392-4f25-4d4a8c1aacaa (at 10.9.104.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4519081800, cur 1561844648 expire 1561844498 last 1561844421 Jun 29 14:44:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 14:49:31 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 14:49:31 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 29 14:59:32 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 14:59:32 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jun 29 15:09:32 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 15:09:32 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 407 previous similar messages Jun 29 15:19:33 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 15:19:33 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 416 previous similar messages Jun 29 15:29:34 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 15:29:34 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 396 previous similar messages Jun 29 15:39:35 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 15:39:35 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 436 previous similar messages Jun 29 15:49:35 fir-md1-s1 kernel: LustreError: 44038:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 15:49:35 fir-md1-s1 kernel: LustreError: 44038:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 416 previous similar messages Jun 29 15:59:35 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 15:59:35 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 412 previous similar messages Jun 29 16:09:37 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 16:09:37 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 414 previous similar messages Jun 29 16:19:37 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 16:19:37 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 439 previous similar messages Jun 29 16:29:38 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 16:29:38 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 446 previous similar messages Jun 29 16:39:39 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 16:39:39 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 456 previous similar messages Jun 29 16:49:40 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 16:49:40 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 453 previous similar messages Jun 29 16:59:41 fir-md1-s1 kernel: LustreError: 46552:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 16:59:41 fir-md1-s1 kernel: LustreError: 46552:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 446 previous similar messages Jun 29 17:09:41 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 17:09:41 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jun 29 17:19:42 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 17:19:42 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 474 previous similar messages Jun 29 17:29:43 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 17:29:43 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 475 previous similar messages Jun 29 17:39:43 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 17:39:43 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 477 previous similar messages Jun 29 17:49:43 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 17:49:43 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 29 17:59:43 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 17:59:43 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 29 18:09:44 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 18:09:44 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 474 previous similar messages Jun 29 18:19:44 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 18:19:44 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 470 previous similar messages Jun 29 18:29:46 fir-md1-s1 kernel: LustreError: 21565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 18:29:46 fir-md1-s1 kernel: LustreError: 21565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 29 18:39:46 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 18:39:46 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 29 18:49:46 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 18:49:46 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jun 29 18:59:46 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 18:59:46 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 467 previous similar messages Jun 29 19:09:47 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 19:09:47 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 436 previous similar messages Jun 29 19:19:47 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 19:19:47 fir-md1-s1 kernel: LustreError: 25632:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 440 previous similar messages Jun 29 19:29:48 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 19:29:48 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 418 previous similar messages Jun 29 19:39:50 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 19:39:50 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 383 previous similar messages Jun 29 19:49:50 fir-md1-s1 kernel: LustreError: 46563:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 19:49:50 fir-md1-s1 kernel: LustreError: 46563:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 416 previous similar messages Jun 29 19:59:51 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 19:59:51 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 390 previous similar messages Jun 29 20:09:52 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 20:09:52 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 371 previous similar messages Jun 29 20:19:53 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 20:19:53 fir-md1-s1 kernel: LustreError: 21294:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 372 previous similar messages Jun 29 20:29:54 fir-md1-s1 kernel: LustreError: 44040:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 20:29:54 fir-md1-s1 kernel: LustreError: 44040:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 397 previous similar messages Jun 29 20:39:55 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 20:39:55 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 425 previous similar messages Jun 29 20:49:55 fir-md1-s1 kernel: LustreError: 46564:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 20:49:55 fir-md1-s1 kernel: LustreError: 46564:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 440 previous similar messages Jun 29 20:59:56 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 20:59:56 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 485 previous similar messages Jun 29 21:09:57 fir-md1-s1 kernel: LustreError: 46564:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 21:09:57 fir-md1-s1 kernel: LustreError: 46564:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 480 previous similar messages Jun 29 21:14:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d9a5b23f-bd1d-b214-10be-ab41be0e273e (at 10.9.108.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f252426c000, cur 1561868065 expire 1561867915 last 1561867838 Jun 29 21:14:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 29 21:14:27 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 5c0904b9-a746-baa3-6518-92bf7219376b (at 10.9.108.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2533274c00, cur 1561868067 expire 1561867917 last 1561867840 Jun 29 21:19:57 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 21:19:57 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jun 29 21:29:59 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 21:29:59 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 449 previous similar messages Jun 29 21:40:00 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 21:40:00 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 440 previous similar messages Jun 29 21:50:00 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 21:50:00 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 442 previous similar messages Jun 29 22:00:00 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 22:00:00 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 431 previous similar messages Jun 29 22:10:01 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 22:10:01 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 419 previous similar messages Jun 29 22:20:01 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 22:20:01 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 438 previous similar messages Jun 29 22:30:01 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 22:30:01 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 446 previous similar messages Jun 29 22:40:02 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 22:40:02 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 456 previous similar messages Jun 29 22:50:02 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 22:50:02 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 453 previous similar messages Jun 29 23:00:03 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 23:00:03 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 436 previous similar messages Jun 29 23:10:03 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 23:10:03 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 438 previous similar messages Jun 29 23:20:04 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 23:20:04 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 435 previous similar messages Jun 29 23:30:04 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 23:30:04 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 438 previous similar messages Jun 29 23:39:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 7effe21a-1be3-b078-9c02-424c4e1d26a9 (at 10.9.106.70@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2521a47000, cur 1561876792 expire 1561876642 last 1561876565 Jun 29 23:39:52 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 29 23:40:04 fir-md1-s1 kernel: LustreError: 44040:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 23:40:04 fir-md1-s1 kernel: LustreError: 44040:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 478 previous similar messages Jun 29 23:50:05 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jun 29 23:50:05 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 441 previous similar messages Jun 30 00:42:50 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 30 00:42:50 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jun 30 00:44:15 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jun 30 00:44:15 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 30 00:46:48 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jun 30 00:46:48 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 6 previous similar messages Jun 30 00:51:52 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jun 30 00:51:52 fir-md1-s1 kernel: LustreError: 21497:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 11 previous similar messages Jun 30 01:11:01 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 30 01:11:01 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7 previous similar messages Jun 30 01:21:16 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jun 30 01:21:16 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 25 previous similar messages Jun 30 01:39:19 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jun 30 01:39:19 fir-md1-s1 kernel: LustreError: 27585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jun 30 01:49:20 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 30 01:49:20 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 41 previous similar messages Jun 30 02:06:37 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jun 30 02:06:37 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7 previous similar messages Jun 30 02:23:16 fir-md1-s1 kernel: LustreError: 21744:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jun 30 02:23:16 fir-md1-s1 kernel: LustreError: 21744:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 69 previous similar messages Jun 30 02:33:35 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 30 02:33:35 fir-md1-s1 kernel: LustreError: 21516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 236 previous similar messages Jun 30 02:43:38 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 02:43:38 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 186 previous similar messages Jun 30 03:00:12 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jun 30 03:00:12 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 31 previous similar messages Jun 30 03:10:21 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 30 03:10:21 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 178 previous similar messages Jun 30 03:20:26 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 03:20:26 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 33 previous similar messages Jun 30 03:48:55 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 03:48:55 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 43 previous similar messages Jun 30 03:52:10 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 03:52:10 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 04:10:35 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 04:13:48 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 04:13:48 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 04:31:56 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 04:32:21 fir-md1-s1 kernel: LustreError: 46528:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 05:42:43 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jun 30 05:42:43 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jun 30 05:42:44 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 30 05:42:44 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 31 previous similar messages Jun 30 05:42:47 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 30 05:42:47 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jun 30 05:43:12 fir-md1-s1 kernel: LustreError: 46579:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 05:43:12 fir-md1-s1 kernel: LustreError: 46579:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jun 30 05:43:22 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 05:43:22 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 26 previous similar messages Jun 30 05:43:41 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jun 30 05:43:41 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 26 previous similar messages Jun 30 05:44:20 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 30 05:44:20 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 179 previous similar messages Jun 30 05:45:50 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jun 30 05:45:50 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 40 previous similar messages Jun 30 05:49:18 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jun 30 05:49:18 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 65 previous similar messages Jun 30 05:54:47 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 155648 GRANT, real grant 0 Jun 30 05:54:47 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 39 previous similar messages Jun 30 06:06:11 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jun 30 06:06:11 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 373 previous similar messages Jun 30 06:16:14 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 06:16:14 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jun 30 06:29:04 fir-md1-s1 kernel: LustreError: 21565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 06:29:04 fir-md1-s1 kernel: LustreError: 21565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jun 30 06:39:31 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 06:39:31 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jun 30 06:49:34 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 06:49:34 fir-md1-s1 kernel: LustreError: 27602:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 18 previous similar messages Jun 30 07:00:03 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 07:00:03 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 130 previous similar messages Jun 30 07:10:07 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 07:10:07 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 95 previous similar messages Jun 30 07:22:13 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 07:22:13 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 39 previous similar messages Jun 30 07:35:01 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 07:45:41 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 07:45:41 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 59 previous similar messages Jun 30 07:56:34 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 07:56:34 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jun 30 08:11:33 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 08:11:33 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jun 30 08:30:56 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 08:30:56 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 30 08:44:10 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 08:44:10 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 30 08:56:08 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 08:56:08 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 6 previous similar messages Jun 30 09:11:21 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 09:11:21 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 09:36:49 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 09:36:49 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jun 30 09:41:52 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 09:41:52 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 09:46:23 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 09:46:23 fir-md1-s1 kernel: LustreError: 21449:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 30 10:01:28 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 10:16:25 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 10:16:25 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7 previous similar messages Jun 30 10:29:20 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 10:46:25 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 10:46:25 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jun 30 11:11:12 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 11:11:12 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 30 11:16:34 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 11:16:34 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 11:22:16 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 11:22:16 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jun 30 11:31:27 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 11:31:27 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 10 previous similar messages Jun 30 11:58:12 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 11:58:12 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 63 previous similar messages Jun 30 12:10:10 fir-md1-s1 kernel: LustreError: 21740:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 12:10:10 fir-md1-s1 kernel: LustreError: 21740:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 12:10:31 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 155648 GRANT, real grant 0 Jun 30 12:23:14 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 12:23:14 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jun 30 12:23:19 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 12:23:19 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jun 30 12:23:24 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 12:23:24 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jun 30 12:23:58 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 12:23:58 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 10 previous similar messages Jun 30 12:24:19 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 12:24:19 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jun 30 12:24:59 fir-md1-s1 kernel: LustreError: 46550:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 12:24:59 fir-md1-s1 kernel: LustreError: 46550:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jun 30 12:26:14 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 12:26:14 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 40 previous similar messages Jun 30 12:28:54 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 32768 GRANT, real grant 0 Jun 30 12:28:54 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 93 previous similar messages Jun 30 12:34:15 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 12:34:15 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 122 previous similar messages Jun 30 13:52:28 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 13:52:28 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jun 30 13:53:43 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 13:53:43 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 18 previous similar messages Jun 30 13:56:24 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 13:56:24 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 43 previous similar messages Jun 30 14:01:40 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 14:01:40 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 77 previous similar messages Jun 30 14:11:54 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 14:11:54 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 94 previous similar messages Jun 30 14:22:44 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 32768 GRANT, real grant 0 Jun 30 14:22:44 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 29 previous similar messages Jun 30 14:32:49 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 14:32:49 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 29 previous similar messages Jun 30 14:42:51 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 14:42:51 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 40 previous similar messages Jun 30 14:53:10 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 14:53:10 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 74 previous similar messages Jun 30 15:03:19 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 15:03:19 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 49 previous similar messages Jun 30 15:13:58 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 79e647da-c3f6-a3be-d8fe-44afe2c61e65 claims 28672 GRANT, real grant 0 Jun 30 15:13:58 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 37 previous similar messages Jun 30 15:24:07 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 5f763aae-b29d-37fb-3cb6-92e44ca397c9 claims 28672 GRANT, real grant 0 Jun 30 15:24:07 fir-md1-s1 kernel: LustreError: 22730:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 30 previous similar messages Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 4c6d21f6-3e09-6b98-bf50-a29faf23fa85 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1cf169c800, cur 1561933606 expire 1561933456 last 1561933379 Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b91c484f-e487-3764-dc4b-13ef610a985a (at 10.8.26.10@o2ib6) reconnecting Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 83e53a59-cc28-333f-3bd7-6445a9dc9fd5 (at 10.8.18.28@o2ib6) Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.16@o2ib6, removing former export from same NID Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d072205a-1b1b-636c-7696-e9d92af1edee (at 10.8.20.3@o2ib6) reconnecting Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1c6cf01f-00af-d021-7941-fb8c37d4ff7c (at 10.8.20.3@o2ib6) Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.2.17@o2ib6, removing former export from same NID Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jun 30 15:26:46 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jun 30 15:26:47 fir-md1-s1 kernel: Lustre: 97638:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561933599/real 0] req@ffff8f207d110c00 x1636719279079584/t0(0) o104->fir-MDT0002@10.8.17.24@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561933606 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:26:47 fir-md1-s1 kernel: Lustre: 97638:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 30 15:26:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.26@o2ib6, removing former export from same NID Jun 30 15:26:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 08a93feb-23b2-0c44-b594-18b6878dec21 (at 10.8.30.18@o2ib6) reconnecting Jun 30 15:26:47 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jun 30 15:26:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 9b97a58f-573c-46e2-b9e4-530918c27ae7 (at 10.8.30.18@o2ib6) Jun 30 15:26:47 fir-md1-s1 kernel: Lustre: Skipped 142 previous similar messages Jun 30 15:26:47 fir-md1-s1 kernel: LustreError: 44034:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1db41ba450 x1635086943010656/t0(0) o4->fe414b50-a889-d1c3-c193-5f58a4966fe7@10.8.1.8@o2ib6:29/0 lens 488/448 e 1 to 0 dl 1561933619 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:26:47 fir-md1-s1 kernel: LustreError: 44034:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 17 previous similar messages Jun 30 15:26:47 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jun 30 15:26:48 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:26:48 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1788eaca00 Jun 30 15:26:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with fe414b50-a889-d1c3-c193-5f58a4966fe7 (at 10.8.1.8@o2ib6), client will retry: rc = -110 Jun 30 15:26:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.31.10@o2ib6, removing former export from same NID Jun 30 15:26:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 19f53614-4ac3-8e1e-1ec0-b9833a2b383f (at 10.8.22.19@o2ib6) reconnecting Jun 30 15:26:49 fir-md1-s1 kernel: Lustre: Skipped 191 previous similar messages Jun 30 15:26:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1835b66b-5b94-b0fe-f70f-6ec070b9ba03 (at 10.8.22.19@o2ib6) Jun 30 15:26:49 fir-md1-s1 kernel: Lustre: Skipped 289 previous similar messages Jun 30 15:26:49 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jun 30 15:26:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 018b4088-9100-7f5b-2709-38dd7f461ac7 (at 10.8.8.29@o2ib6) reconnecting Jun 30 15:26:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to c952479e-eacc-a158-a9f3-c256f0987c93 (at 10.8.23.32@o2ib6) Jun 30 15:26:53 fir-md1-s1 kernel: Lustre: Skipped 430 previous similar messages Jun 30 15:26:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.23.32@o2ib6, removing former export from same NID Jun 30 15:26:53 fir-md1-s1 kernel: Lustre: Skipped 138 previous similar messages Jun 30 15:26:53 fir-md1-s1 kernel: Lustre: Skipped 290 previous similar messages Jun 30 15:26:54 fir-md1-s1 kernel: Lustre: 24585:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f15b05dfb00 x1631310231321104/t0(0) o101->2defae61-8bf0-dee6-7d48-53b83a69e973@10.8.17.24@o2ib6:29/0 lens 480/568 e 1 to 0 dl 1561933619 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:27:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client d3e22dd2-d25d-28e8-5f86-5d27043eaa8d (at 10.8.7.18@o2ib6) reconnecting Jun 30 15:27:01 fir-md1-s1 kernel: Lustre: Skipped 461 previous similar messages Jun 30 15:27:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.23.9@o2ib6, removing former export from same NID Jun 30 15:27:01 fir-md1-s1 kernel: Lustre: Skipped 230 previous similar messages Jun 30 15:27:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 0bcd0825-5f35-e709-e57c-d41ae345f214 (at 10.8.23.9@o2ib6) Jun 30 15:27:01 fir-md1-s1 kernel: Lustre: Skipped 695 previous similar messages Jun 30 15:27:02 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 5cad2422-3e98-66d4-e9e4-0ce15d870f56 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0d62f99800, cur 1561933622 expire 1561933472 last 1561933395 Jun 30 15:27:06 fir-md1-s1 kernel: Lustre: 44034:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1f5620a850 x1631558639360624/t0(0) o4->84fd8c4b-6545-cd41-282d-ef5f651cba30@10.8.17.11@o2ib6:11/0 lens 488/448 e 1 to 0 dl 1561933631 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:27:09 fir-md1-s1 kernel: LustreError: 46560:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1f0f920050 x1634321379548032/t0(0) o4->545f12c1-4799-a254-b9c4-f75f43e1bc5b@10.8.27.23@o2ib6:26/0 lens 488/448 e 1 to 0 dl 1561933646 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:27:09 fir-md1-s1 kernel: LustreError: 46560:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message Jun 30 15:27:19 fir-md1-s1 kernel: Lustre: 22005:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561933632/real 0] req@ffff8f19b179f200 x1636719279124976/t0(0) o104->fir-MDT0002@10.8.8.37@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561933639 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:27:20 fir-md1-s1 kernel: Lustre: 97638:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:21s); client may timeout. req@ffff8f15b05dfb00 x1631310231321104/t0(0) o101->2defae61-8bf0-dee6-7d48-53b83a69e973@10.8.17.24@o2ib6:29/0 lens 480/536 e 1 to 0 dl 1561933619 ref 1 fl Complete:/0/0 rc 0/0 Jun 30 15:27:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 7b8c2334-5441-fafb-761f-7bfdc2fe1e61 (at 10.8.18.30@o2ib6) reconnecting Jun 30 15:27:20 fir-md1-s1 kernel: Lustre: Skipped 302 previous similar messages Jun 30 15:27:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 2ded1f4a-314b-a6a0-d3a0-8acbcea0369c (at 10.8.18.30@o2ib6) Jun 30 15:27:20 fir-md1-s1 kernel: Lustre: Skipped 459 previous similar messages Jun 30 15:27:21 fir-md1-s1 kernel: Lustre: 25635:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1f0f920050 x1634321379548032/t0(0) o4->545f12c1-4799-a254-b9c4-f75f43e1bc5b@10.8.27.23@o2ib6:26/0 lens 488/448 e 1 to 0 dl 1561933646 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:27:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.15.3@o2ib6, removing former export from same NID Jun 30 15:27:22 fir-md1-s1 kernel: Lustre: Skipped 154 previous similar messages Jun 30 15:27:27 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:27:27 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1f3983ec00 Jun 30 15:27:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6), client will retry: rc = -110 Jun 30 15:27:27 fir-md1-s1 kernel: Lustre: 21516:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:16s); client may timeout. req@ffff8f1f5620a850 x1631558639360624/t0(0) o4->84fd8c4b-6545-cd41-282d-ef5f651cba30@10.8.17.11@o2ib6:11/0 lens 488/448 e 1 to 0 dl 1561933631 ref 1 fl Complete:/0/ffffffff rc -110/-1 Jun 30 15:27:28 fir-md1-s1 kernel: Lustre: 22287:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1bf1f95400 x1634927464071184/t349380999792(0) o36->a2d1cfa6-4e2d-7226-3700-dc24c44c8e97@10.9.108.16@o2ib4:2/0 lens 488/3152 e 1 to 0 dl 1561933652 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:27:35 fir-md1-s1 kernel: Lustre: 20458:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561933648/real 0] req@ffff8f0df9831e00 x1636719279145264/t0(0) o104->fir-MDT0002@10.8.8.37@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561933655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:27:41 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.8.37@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f24971e3a80/0x5d9ee62ab146b59d lrc: 4/0,0 mode: PR/PR res: [0x2c002c268:0x189:0x0].0x0 bits 0x1b/0x0 rrc: 14 type: IBT flags: 0x60200400000020 nid: 10.8.8.37@o2ib6 remote: 0x7c2525caa0dc4085 expref: 4991 pid: 97662 timeout: 1048721 lvb_type: 0 Jun 30 15:27:41 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 30 15:27:43 fir-md1-s1 kernel: Lustre: 23602:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0545373f00 x1631546446279584/t349381007355(0) o36->25c05458-1ff8-5b3c-505b-360943a414ba@10.9.104.66@o2ib4:18/0 lens 488/3152 e 1 to 0 dl 1561933668 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:27:47 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:27:47 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e7bba7c00 Jun 30 15:27:47 fir-md1-s1 kernel: Lustre: 46560:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:21s); client may timeout. req@ffff8f1f0f920050 x1634321379548032/t0(0) o4->545f12c1-4799-a254-b9c4-f75f43e1bc5b@10.8.27.23@o2ib6:26/0 lens 488/448 e 1 to 0 dl 1561933646 ref 1 fl Complete:/0/ffffffff rc -110/-1 Jun 30 15:27:48 fir-md1-s1 kernel: Lustre: 20378:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561933661/real 0] req@ffff8f19b179da00 x1636719279156688/t0(0) o104->fir-MDT0002@10.8.0.68@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561933668 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:27:48 fir-md1-s1 kernel: Lustre: 20378:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 30 15:27:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 8ce01d8b-55b4-edf0-189b-0eb92aac6c18 (at 10.8.21.11@o2ib6) Jun 30 15:27:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 486bedd2-dd65-4f17-854a-54ed20ee472c (at 10.8.21.11@o2ib6) reconnecting Jun 30 15:27:52 fir-md1-s1 kernel: Lustre: Skipped 1103 previous similar messages Jun 30 15:27:52 fir-md1-s1 kernel: Lustre: Skipped 1660 previous similar messages Jun 30 15:27:53 fir-md1-s1 kernel: Lustre: 23638:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f396a602d00 x1635088425115888/t349381010483(0) o36->9c7adb50-64f1-6d92-d619-cdf901757223@10.9.108.11@o2ib4:28/0 lens 488/3152 e 1 to 0 dl 1561933678 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:27:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.22@o2ib6, removing former export from same NID Jun 30 15:27:54 fir-md1-s1 kernel: Lustre: Skipped 601 previous similar messages Jun 30 15:28:13 fir-md1-s1 kernel: Lustre: 23613:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f17cb07d100 x1631598529828544/t0(0) o101->7f8dc145-a081-da87-1da4-154358301486@10.9.108.1@o2ib4:18/0 lens 576/3264 e 1 to 0 dl 1561933698 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:28:13 fir-md1-s1 kernel: Lustre: 23613:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages Jun 30 15:28:19 fir-md1-s1 kernel: LustreError: 23682:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.0.65@o2ib6) failed to reply to blocking AST (req@ffff8f3afd38c200 x1636719279149952 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f22e3389f80/0x5d9ee62aa31edf45 lrc: 4/0,0 mode: PR/PR res: [0x200029c3c:0xc95:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.0.65@o2ib6 remote: 0xf6c4443c4931009d expref: 510356 pid: 21483 timeout: 1048741 lvb_type: 0 Jun 30 15:28:19 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.0.65@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jun 30 15:28:40 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.0.68@o2ib6) failed to reply to blocking AST (req@ffff8f19b179da00 x1636719279156688 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f1ee118e780/0x5d9ee62aad882d3e lrc: 4/0,0 mode: PR/PR res: [0x2c002bff7:0x491:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x60000400010020 nid: 10.8.0.68@o2ib6 remote: 0xe8ef120cd2738cc3 expref: 29209 pid: 22280 timeout: 1048751 lvb_type: 0 Jun 30 15:28:40 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.0.68@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jun 30 15:28:40 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 59s: evicting client at 10.8.0.68@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1ee118e780/0x5d9ee62aad882d3e lrc: 3/0,0 mode: PR/PR res: [0x2c002bff7:0x491:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x60000400010020 nid: 10.8.0.68@o2ib6 remote: 0xe8ef120cd2738cc3 expref: 29210 pid: 22280 timeout: 0 lvb_type: 0 Jun 30 15:28:46 fir-md1-s1 kernel: Lustre: 97664:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-26), not sending early reply req@ffff8f1ebd0add00 x1635088425129824/t0(0) o101->9c7adb50-64f1-6d92-d619-cdf901757223@10.9.108.11@o2ib4:21/0 lens 576/3264 e 0 to 0 dl 1561933731 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:28:46 fir-md1-s1 kernel: Lustre: 97664:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 35 previous similar messages Jun 30 15:28:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client ac9cd631-a534-1fba-753c-5069b079d1ad (at 10.8.24.16@o2ib6) reconnecting Jun 30 15:28:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5afbb881-550f-fa08-cafd-4158b37c9811 (at 10.8.24.16@o2ib6) Jun 30 15:28:57 fir-md1-s1 kernel: Lustre: Skipped 2584 previous similar messages Jun 30 15:28:57 fir-md1-s1 kernel: Lustre: Skipped 1734 previous similar messages Jun 30 15:28:59 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.7.35@o2ib6, removing former export from same NID Jun 30 15:28:59 fir-md1-s1 kernel: Lustre: Skipped 807 previous similar messages Jun 30 15:29:08 fir-md1-s1 kernel: LustreError: 25677:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f42b3bc9b00 x1636719279237744/t0(0) o104->fir-MDT0000@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 15:29:37 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.0.65@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1eb9c7af40/0x5d9ee62aabaa4472 lrc: 3/0,0 mode: PR/PR res: [0x200029c3c:0xde5:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.0.65@o2ib6 remote: 0xf6c4443c496a0e5c expref: 269628 pid: 97672 timeout: 1048837 lvb_type: 0 Jun 30 15:29:37 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 30 15:29:38 fir-md1-s1 kernel: Lustre: 23683:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561933771/real 0] req@ffff8f1329df7800 x1636719279282784/t0(0) o104->fir-MDT0000@10.8.0.66@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561933778 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:29:56 fir-md1-s1 kernel: Lustre: 23594:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f14a4515100 x1634478624590720/t0(0) o101->e15f364b-b556-833b-9c7c-0e0e1407bf82@10.9.0.62@o2ib4:1/0 lens 1776/3288 e 0 to 0 dl 1561933801 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:29:56 fir-md1-s1 kernel: Lustre: 23594:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 28 previous similar messages Jun 30 15:30:00 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.15.3@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f18c074e0c0/0x5d9ee62ab48a8a40 lrc: 4/0,0 mode: PR/PR res: [0x200029c4a:0xb328:0x0].0x0 bits 0x13/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 10.8.15.3@o2ib6 remote: 0x6292ab32aa6537dd expref: 11526 pid: 97663 timeout: 1048860 lvb_type: 0 Jun 30 15:30:04 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:30:19 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.17.12@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1be49a4800/0x5d9ee62ab0affc4d lrc: 4/0,0 mode: PR/PR res: [0x2c002c309:0xc4a1:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x60000400010020 nid: 10.8.17.12@o2ib6 remote: 0xb9a0d20b4227c5b2 expref: 5562 pid: 97664 timeout: 1048879 lvb_type: 0 Jun 30 15:30:19 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 30 15:30:38 fir-md1-s1 kernel: LustreError: 25677:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561933748, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f450566d580/0x5d9ee62ab5001b5a lrc: 3/0,1 mode: --/CW res: [0x200029c3c:0xde5:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 25677 timeout: 0 lvb_type: 0 Jun 30 15:31:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client d22c54d9-f3ee-e6f8-f34c-cd9ceccbd787 (at 10.8.2.24@o2ib6) reconnecting Jun 30 15:31:05 fir-md1-s1 kernel: Lustre: Skipped 2866 previous similar messages Jun 30 15:31:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 721a064b-1659-db7a-cc36-f67cf8b564bb (at 10.8.2.24@o2ib6) Jun 30 15:31:05 fir-md1-s1 kernel: Lustre: Skipped 4302 previous similar messages Jun 30 15:31:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.15.9@o2ib6, removing former export from same NID Jun 30 15:31:12 fir-md1-s1 kernel: Lustre: Skipped 1419 previous similar messages Jun 30 15:31:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 4343a906-23d9-f729-b768-bcd0549ada0d (at 10.8.8.37@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1e67882400, cur 1561933891 expire 1561933741 last 1561933664 Jun 30 15:31:38 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:32:13 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:32:14 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561933927/real 0] req@ffff8f161d74bc00 x1636719279463696/t0(0) o106->fir-MDT0000@10.8.18.14@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1561933934 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:32:14 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jun 30 15:32:38 fir-md1-s1 kernel: Lustre: 22289:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2496f3b600 x1631538936016256/t0(0) o101->769d013d-f990-3399-dde8-f67f737a957d@10.8.7.25@o2ib6:13/0 lens 576/3264 e 1 to 0 dl 1561933963 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:32:38 fir-md1-s1 kernel: Lustre: 22289:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jun 30 15:32:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 30 15:32:41 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 30 15:32:46 fir-md1-s1 kernel: LustreError: 20730:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f251a7be000 x1636719279554688/t0(0) o104->fir-MDT0000@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 15:32:46 fir-md1-s1 kernel: Lustre: 21456:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:3s); client may timeout. req@ffff8f2496f3b600 x1631538936016256/t0(0) o101->769d013d-f990-3399-dde8-f67f737a957d@10.8.7.25@o2ib6:13/0 lens 576/536 e 1 to 0 dl 1561933963 ref 1 fl Complete:/0/0 rc 0/0 Jun 30 15:33:16 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:33:16 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages Jun 30 15:33:23 fir-md1-s1 kernel: Lustre: 21483:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561933996/real 0] req@ffff8f161aaf9e00 x1636719279725872/t0(0) o104->fir-MDT0002@10.8.28.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561934003 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:33:23 fir-md1-s1 kernel: Lustre: 21483:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 30 15:33:46 fir-md1-s1 kernel: LustreError: 46576:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f204e271450 x1636413998790944/t0(0) o4->f5114f0b-b017-9912-d44d-f24fe0d2ebc9@10.8.26.33@o2ib6:22/0 lens 488/448 e 1 to 0 dl 1561934032 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:33:59 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:33:59 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2055541a00 Jun 30 15:33:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with f5114f0b-b017-9912-d44d-f24fe0d2ebc9 (at 10.8.26.33@o2ib6), client will retry: rc = -110 Jun 30 15:33:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 30 15:33:59 fir-md1-s1 kernel: Lustre: 46576:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:7s); client may timeout. req@ffff8f204e271450 x1636413998790944/t0(0) o4->f5114f0b-b017-9912-d44d-f24fe0d2ebc9@10.8.26.33@o2ib6:22/0 lens 488/448 e 1 to 0 dl 1561934032 ref 1 fl Complete:/0/ffffffff rc -110/-1 Jun 30 15:34:03 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20fb121200 Jun 30 15:34:03 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1ce3c76400 Jun 30 15:34:07 fir-md1-s1 kernel: LustreError: 55488:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 30+0s req@ffff8f4503e59c50 x1636439708308352/t0(0) o256->420c129b-df9e-b1c5-eae5-667fed64bb9d@10.8.15.3@o2ib6:7/0 lens 304/240 e 0 to 0 dl 1561934047 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:34:07 fir-md1-s1 kernel: LustreError: 55488:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message Jun 30 15:34:10 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f3a50f0e000 Jun 30 15:34:11 fir-md1-s1 kernel: LustreError: 55538:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 30+0s req@ffff8f1f5dab7050 x1637349789347104/t0(0) o256->e02527c5-320a-cc02-89d1-b5d3560ed7b2@10.8.0.67@o2ib6:11/0 lens 304/240 e 0 to 0 dl 1561934051 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:34:11 fir-md1-s1 kernel: LustreError: 55538:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 11 previous similar messages Jun 30 15:34:11 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f16bccd1e00 Jun 30 15:34:12 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e022fc400 Jun 30 15:34:12 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f18b7296e00 Jun 30 15:34:12 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d6ff66000 Jun 30 15:34:13 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1bf0b1a200 Jun 30 15:34:13 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2a146f6400 Jun 30 15:34:13 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f44ff601a00 Jun 30 15:34:15 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f164c3c4c00 Jun 30 15:34:17 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f3492ad9200 Jun 30 15:34:17 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2a146f7400 Jun 30 15:34:18 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f3cf9aa4200 Jun 30 15:34:19 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f26a5e9b400 Jun 30 15:34:19 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1c68f0fe00 Jun 30 15:34:20 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f33e6b8ce00 Jun 30 15:34:20 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f326fa6c600 Jun 30 15:34:21 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1bee721000 Jun 30 15:34:21 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e022ffc00 Jun 30 15:34:21 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f37cd7fb600 Jun 30 15:34:22 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.7.28@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f16b9204140/0x5d9ee62ab52808e9 lrc: 4/0,0 mode: PW/PW res: [0x2c002bf5b:0x10a2:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.8.7.28@o2ib6 remote: 0x4d12180afde56068 expref: 912 pid: 22281 timeout: 1049122 lvb_type: 0 Jun 30 15:34:26 fir-md1-s1 kernel: LustreError: 50447:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f24f48d3400 ns: mdt-fir-MDT0002_UUID lock: ffff8f1888aaf980/0x5d9ee62ab535031f lrc: 3/0,0 mode: PW/PW res: [0x2c002bf5b:0x10a2:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.8.7.28@o2ib6 remote: 0x4d12180afde56a94 expref: 3 pid: 50447 timeout: 0 lvb_type: 0 Jun 30 15:35:04 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:35:04 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 24 previous similar messages Jun 30 15:35:05 fir-md1-s1 kernel: LustreError: 22730:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f1f0f920050 x1637547176047040/t0(0) o4->4c6d21f6-3e09-6b98-bf50-a29faf23fa85@10.8.9.9@o2ib6:5/0 lens 488/448 e 1 to 0 dl 1561934105 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:35:05 fir-md1-s1 kernel: LustreError: 22730:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 4 previous similar messages Jun 30 15:35:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client b16e4006-ad8f-de37-ede7-21e0aff43fcc (at 10.8.1.3@o2ib6) reconnecting Jun 30 15:35:22 fir-md1-s1 kernel: Lustre: Skipped 2840 previous similar messages Jun 30 15:35:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 58f0ff23-7960-f6fe-9a2f-be2834e287bf (at 10.8.1.3@o2ib6) Jun 30 15:35:22 fir-md1-s1 kernel: Lustre: Skipped 3631 previous similar messages Jun 30 15:35:28 fir-md1-s1 kernel: LustreError: 46577:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f1f0f927850 x1631584018818144/t0(0) o4->cc57ad24-07f9-6270-9e45-e86bdff220e7@10.8.2.27@o2ib6:28/0 lens 488/448 e 1 to 0 dl 1561934128 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:35:28 fir-md1-s1 kernel: LustreError: 46577:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message Jun 30 15:35:29 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d78923600 Jun 30 15:35:29 fir-md1-s1 kernel: Lustre: 46577:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:1s); client may timeout. req@ffff8f1f0f927850 x1631584018818144/t0(0) o4->cc57ad24-07f9-6270-9e45-e86bdff220e7@10.8.2.27@o2ib6:28/0 lens 488/448 e 1 to 0 dl 1561934128 ref 1 fl Complete:/0/ffffffff rc -110/-1 Jun 30 15:35:29 fir-md1-s1 kernel: Lustre: 46577:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 20 previous similar messages Jun 30 15:35:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.7.35@o2ib6, removing former export from same NID Jun 30 15:35:46 fir-md1-s1 kernel: Lustre: Skipped 774 previous similar messages Jun 30 15:35:52 fir-md1-s1 kernel: Lustre: 97672:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561934145/real 0] req@ffff8f18afef3300 x1636719279803936/t0(0) o104->fir-MDT0000@10.8.8.18@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561934152 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:35:52 fir-md1-s1 kernel: Lustre: 97672:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jun 30 15:36:10 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 90d881d2-bbfa-565d-91e5-ddef873ff667 (at 10.9.105.48@o2ib4) in 214 seconds. I think it's dead, and I am evicting it. exp ffff8f2523409400, cur 1561934170 expire 1561934020 last 1561933956 Jun 30 15:36:10 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 30 15:36:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 30 15:36:16 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 30 15:36:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client f05bc850-7a22-d5dd-120f-662214ba49f9 (at 10.9.105.48@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fe7cd000, cur 1561934183 expire 1561934033 last 1561933956 Jun 30 15:37:26 fir-md1-s1 kernel: Lustre: 27481:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1db41ba850 x1636416686482832/t0(0) o4->b5d37fef-ba24-e714-aa45-15692218e88e@10.8.1.20@o2ib6:0/0 lens 488/448 e 1 to 0 dl 1561934250 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:37:26 fir-md1-s1 kernel: Lustre: 27481:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 54 previous similar messages Jun 30 15:37:30 fir-md1-s1 kernel: LustreError: 46581:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f1db41ba850 x1636416686482832/t0(0) o4->b5d37fef-ba24-e714-aa45-15692218e88e@10.8.1.20@o2ib6:0/0 lens 488/448 e 1 to 0 dl 1561934250 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:37:43 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:37:43 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 23 previous similar messages Jun 30 15:37:43 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1bba41dc00 Jun 30 15:37:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with b5d37fef-ba24-e714-aa45-15692218e88e (at 10.8.1.20@o2ib6), client will retry: rc = -110 Jun 30 15:37:43 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jun 30 15:37:43 fir-md1-s1 kernel: Lustre: 46581:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:13s); client may timeout. req@ffff8f1db41ba850 x1636416686482832/t0(0) o4->b5d37fef-ba24-e714-aa45-15692218e88e@10.8.1.20@o2ib6:0/0 lens 488/448 e 1 to 0 dl 1561934250 ref 1 fl Complete:/0/ffffffff rc -110/-1 Jun 30 15:37:43 fir-md1-s1 kernel: Lustre: 46581:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jun 30 15:37:45 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.28.11@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f177ff38fc0/0x5d9ee62ab52f055b lrc: 4/0,0 mode: PR/PR res: [0x2c002c279:0x18eb0:0x0].0x0 bits 0x5b/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.28.11@o2ib6 remote: 0xfc47ba20c0093b9a expref: 1111 pid: 97667 timeout: 1049325 lvb_type: 0 Jun 30 15:37:47 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1b5d8d4a00 Jun 30 15:37:56 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1cd30c0a00 Jun 30 15:37:56 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f15e010d600 Jun 30 15:37:56 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2085063000 Jun 30 15:37:56 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1cd30c2800 Jun 30 15:37:56 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f15e010f200 Jun 30 15:38:01 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1ccbbe0e00 Jun 30 15:38:02 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e442abc00 Jun 30 15:38:08 fir-md1-s1 kernel: LustreError: 25085:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.65@o2ib6 arrived at 1561934288 with bad export cookie 6746082362947164325 Jun 30 15:38:08 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f15dc3e8600 Jun 30 15:38:09 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f24cf716e00 Jun 30 15:38:12 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d447fe200 Jun 30 15:38:14 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2008224600 Jun 30 15:38:16 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1b052afc00 Jun 30 15:38:23 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f16a7ab3e00 Jun 30 15:38:24 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.0.68@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f15804d3a80/0x5d9ee62ab55464f6 lrc: 4/0,0 mode: PR/PR res: [0x2c00271dd:0x2c18:0x0].0x0 bits 0x13/0x0 rrc: 161 type: IBT flags: 0x60200400000020 nid: 10.8.0.68@o2ib6 remote: 0xe8ef120cd27956b6 expref: 83 pid: 20462 timeout: 1049364 lvb_type: 0 Jun 30 15:38:24 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 30 15:38:25 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f222cf14600 Jun 30 15:38:26 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1dc52a3a00 Jun 30 15:38:26 fir-md1-s1 kernel: LustreError: 23104:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.68@o2ib6 arrived at 1561934306 with bad export cookie 6746082362948369704 Jun 30 15:38:27 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f247faa7000 Jun 30 15:38:56 fir-md1-s1 kernel: LustreError: 21516:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1f56209850 x1637547176047040/t0(0) o4->4c6d21f6-3e09-6b98-bf50-a29faf23fa85@10.8.9.9@o2ib6:0/0 lens 488/448 e 1 to 0 dl 1561934340 ref 1 fl Interpret:/2/0 rc 0/0 Jun 30 15:38:56 fir-md1-s1 kernel: LustreError: 21516:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 4 previous similar messages Jun 30 15:39:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 30 15:39:05 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 30 15:40:13 fir-md1-s1 kernel: LustreError: 46579:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 30+0s req@ffff8f1f5620ac50 x1631566036542976/t0(0) o4->0e7d6cbd-2dc2-8104-92fb-8187f3b6e75a@10.8.8.11@o2ib6:13/0 lens 488/448 e 0 to 0 dl 1561934413 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:40:13 fir-md1-s1 kernel: LustreError: 46579:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 17 previous similar messages Jun 30 15:40:17 fir-md1-s1 kernel: Lustre: 20729:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1561934405/real 0] req@ffff8f1d73b8ec00 x1636719280117968/t0(0) o104->fir-MDT0000@10.8.18.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561934417 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 15:40:17 fir-md1-s1 kernel: Lustre: 20729:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jun 30 15:40:18 fir-md1-s1 kernel: LustreError: 22289:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.29.3@o2ib6) failed to reply to blocking AST (req@ffff8f1e29c0c800 x1636719280087376 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f18a8227740/0x5d9ee62ab5670d8f lrc: 4/0,0 mode: EX/EX res: [0x2c002be65:0x19553:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT flags: 0x60000400000020 nid: 10.8.29.3@o2ib6 remote: 0x96ea67d338c51e0e expref: 881 pid: 21457 timeout: 1049472 lvb_type: 3 Jun 30 15:40:18 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.29.3@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jun 30 15:40:18 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 36s: evicting client at 10.8.29.3@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f18a8227740/0x5d9ee62ab5670d8f lrc: 3/0,0 mode: EX/EX res: [0x2c002be65:0x19553:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT flags: 0x60000400000020 nid: 10.8.29.3@o2ib6 remote: 0x96ea67d338c51e0e expref: 882 pid: 21457 timeout: 0 lvb_type: 3 Jun 30 15:40:18 fir-md1-s1 kernel: Lustre: 22289:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:6s); client may timeout. req@ffff8f22eea4b300 x1636442429901408/t0(0) o101->9eed212b-34d9-6e26-f1ac-cdc452decf97@10.8.29.3@o2ib6:12/0 lens 376/312 e 0 to 0 dl 1561934412 ref 1 fl Complete:/0/0 rc 0/0 Jun 30 15:40:18 fir-md1-s1 kernel: Lustre: 22289:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 20 previous similar messages Jun 30 15:40:18 fir-md1-s1 kernel: LustreError: 55142:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f098026e300 x1636719280127728/t0(0) o105->fir-MDT0002@10.8.29.3@o2ib6:15/16 lens 304/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 15:40:18 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20b5394000 Jun 30 15:40:20 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f34c4ee7e00 Jun 30 15:40:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 4343a906-23d9-f729-b768-bcd0549ada0d (at 10.8.8.37@o2ib6), client will retry: rc -110 Jun 30 15:40:20 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 30 15:40:22 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f26a5e9c800 Jun 30 15:40:32 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1bdbe50a00 Jun 30 15:40:32 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2029a79200 Jun 30 15:40:32 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1bdbe51200 Jun 30 15:40:32 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f17e18f6e00 Jun 30 15:40:36 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20c4bd2200 Jun 30 15:40:39 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f19ba722a00 Jun 30 15:40:45 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1f45194a00 Jun 30 15:40:45 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e66662600 Jun 30 15:40:45 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e66663200 Jun 30 15:40:45 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1b42657600 Jun 30 15:40:45 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d2f29bc00 Jun 30 15:40:46 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1ee8496e00 Jun 30 15:40:46 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f104436f600 Jun 30 15:40:47 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1dcbfff000 Jun 30 15:40:47 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2509db6e00 Jun 30 15:40:47 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1dcbfff800 Jun 30 15:41:01 fir-md1-s1 kernel: LustreError: 50444:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561934371, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1675ae8fc0/0x5d9ee62ab566aaa1 lrc: 3/0,1 mode: --/PW res: [0x2c002c30a:0xda:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 50444 timeout: 0 lvb_type: 0 Jun 30 15:41:04 fir-md1-s1 kernel: LustreError: 20462:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561934374, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1e433c4a40/0x5d9ee62ab566dc0a lrc: 3/0,1 mode: --/PW res: [0x2c002c2eb:0x31d:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20462 timeout: 0 lvb_type: 0 Jun 30 15:42:00 fir-md1-s1 kernel: LustreError: 50444:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f196e53d800 ns: mdt-fir-MDT0002_UUID lock: ffff8f1675ae8fc0/0x5d9ee62ab566aaa1 lrc: 3/0,0 mode: PW/PW res: [0x2c002c30a:0xda:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x50200000000000 nid: 10.8.9.9@o2ib6 remote: 0x4e1a1dc9d9bcc5a2 expref: 3 pid: 50444 timeout: 0 lvb_type: 0 Jun 30 15:42:06 fir-md1-s1 kernel: LustreError: 20368:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_convert from 10.8.9.9@o2ib6 arrived at 1561934526 with bad export cookie 6746082362948642522 Jun 30 15:42:07 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jun 30 15:42:07 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 55 previous similar messages Jun 30 15:42:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 30 15:42:11 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jun 30 15:43:22 fir-md1-s1 kernel: LustreError: 46579:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f1f5620c450 x1637547176047040/t0(0) o4->4c6d21f6-3e09-6b98-bf50-a29faf23fa85@10.8.9.9@o2ib6:22/0 lens 488/448 e 1 to 0 dl 1561934602 ref 1 fl Interpret:/2/0 rc 0/0 Jun 30 15:43:22 fir-md1-s1 kernel: LustreError: 46579:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 17 previous similar messages Jun 30 15:43:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO write error with 4c6d21f6-3e09-6b98-bf50-a29faf23fa85 (at 10.8.9.9@o2ib6), client will retry: rc = -110 Jun 30 15:43:22 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jun 30 15:44:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 1b90433c-235e-7531-cfe6-8ebc9f785a9b (at 10.9.0.64@o2ib4) reconnecting Jun 30 15:44:10 fir-md1-s1 kernel: Lustre: Skipped 176 previous similar messages Jun 30 15:44:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b6973ee9-fc9f-d2c6-1102-75dfdfeafb62 (at 10.9.0.64@o2ib4) Jun 30 15:44:10 fir-md1-s1 kernel: Lustre: Skipped 211 previous similar messages Jun 30 15:44:17 fir-md1-s1 kernel: LustreError: 27580:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8f1f5620ac50 x1634457905713520/t0(0) o3->1e4e71a5-88c6-3b3c-8591-f6af96f4c86f@10.8.1.28@o2ib6:16/0 lens 488/440 e 0 to 0 dl 1561934686 ref 1 fl Interpret:/0/0 rc 0/0 Jun 30 15:44:17 fir-md1-s1 kernel: LustreError: 27580:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message Jun 30 15:44:23 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1c0f854e00 Jun 30 15:44:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 1e4e71a5-88c6-3b3c-8591-f6af96f4c86f (at 10.8.1.28@o2ib6), client will retry: rc -110 Jun 30 15:44:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.9.9@o2ib6, removing former export from same NID Jun 30 15:44:31 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jun 30 15:44:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 30 15:44:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 30 15:45:17 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1dcd61d580/0x5d9ee62ab47759e4 lrc: 3/0,0 mode: PR/PR res: [0x200025db9:0x1e79:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x4e1a1dc9d9463516 expref: 511061 pid: 24585 timeout: 1049777 lvb_type: 0 Jun 30 15:45:17 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 30 15:45:34 fir-md1-s1 kernel: LustreError: 50444:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1f49ea8c00 x1636719280618928/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 15:45:59 fir-md1-s1 kernel: Lustre: 20462:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1f30fad100 x1637547237343424/t0(0) o101->4c6d21f6-3e09-6b98-bf50-a29faf23fa85@10.8.9.9@o2ib6:4/0 lens 480/568 e 0 to 0 dl 1561934764 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 15:45:59 fir-md1-s1 kernel: Lustre: 20462:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 100 previous similar messages Jun 30 15:46:18 fir-md1-s1 kernel: LustreError: 50447:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561934688, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f16f47f9f80/0x5d9ee62ab5a2e4bb lrc: 3/0,1 mode: --/CW res: [0x200025db9:0x1e79:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 50447 timeout: 0 lvb_type: 0 Jun 30 15:46:29 fir-md1-s1 kernel: LustreError: 97672:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f20e276a700 x1636719280693616/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 15:46:29 fir-md1-s1 kernel: LustreError: 97672:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Jun 30 15:47:04 fir-md1-s1 kernel: LustreError: 50444:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561934734, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f24067ee780/0x5d9ee62ab5b5a5bf lrc: 3/0,1 mode: --/PW res: [0x200029c58:0x8a1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 50444 timeout: 0 lvb_type: 0 Jun 30 15:47:21 fir-md1-s1 kernel: LustreError: 50444:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1f49eab000 x1636719280753872/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 15:47:59 fir-md1-s1 kernel: LustreError: 97672:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561934789, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1605a0c140/0x5d9ee62ab5f15d64 lrc: 3/0,1 mode: --/PW res: [0x200029c58:0x894:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97672 timeout: 0 lvb_type: 0 Jun 30 15:48:51 fir-md1-s1 kernel: LustreError: 50444:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561934841, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f163ae4f740/0x5d9ee62ab60ba639 lrc: 3/0,1 mode: --/PW res: [0x200029c58:0x8a0:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 50444 timeout: 0 lvb_type: 0 Jun 30 16:15:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 4c6d21f6-3e09-6b98-bf50-a29faf23fa85 (at 10.8.9.9@o2ib6) reconnecting Jun 30 16:15:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.9.9@o2ib6, removing former export from same NID Jun 30 16:15:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 30 16:15:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 30 16:15:47 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jun 30 16:15:47 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jun 30 16:18:42 fir-md1-s1 kernel: Lustre: 21481:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f18456b5d00 x1636442616557568/t0(0) o101->9eed212b-34d9-6e26-f1ac-cdc452decf97@10.8.29.3@o2ib6:17/0 lens 480/568 e 1 to 0 dl 1561936727 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 16:18:42 fir-md1-s1 kernel: Lustre: 21481:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Jun 30 16:18:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 9eed212b-34d9-6e26-f1ac-cdc452decf97 (at 10.8.29.3@o2ib6) reconnecting Jun 30 16:18:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 86af1d07-9f84-ff94-71a6-68fd12f8c1ac (at 10.8.29.3@o2ib6) Jun 30 16:18:48 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jun 30 16:18:51 fir-md1-s1 kernel: Lustre: 22289:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561936722/real 1561936722] req@ffff8f1a1da4dd00 x1636719283776080/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561936731 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 16:18:51 fir-md1-s1 kernel: Lustre: 22289:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jun 30 18:19:02 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 584e20c4-52de-2973-da1d-0e2ebca7e50e (at 10.9.104.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1465b3f800, cur 1561943942 expire 1561943792 last 1561943715 Jun 30 18:19:02 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 30 18:19:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 4120f4aa-15d3-15d6-3436-73087cc4dacd (at 10.9.104.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0a8b296c00, cur 1561943947 expire 1561943797 last 1561943720 Jun 30 18:19:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 30 18:35:02 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jun 30 18:35:02 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jun 30 18:36:27 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jun 30 18:36:27 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 100 previous similar messages Jun 30 18:39:28 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jun 30 18:39:28 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jun 30 18:48:23 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 32768 GRANT, real grant 0 Jun 30 18:48:23 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jun 30 19:20:53 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jun 30 19:20:53 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 19:25:34 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jun 30 19:38:33 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jun 30 19:38:55 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 19:38:55 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22 previous similar messages Jun 30 19:39:36 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 19:39:36 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 74 previous similar messages Jun 30 19:44:00 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 19:44:00 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7 previous similar messages Jun 30 19:54:58 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jun 30 19:54:58 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 21:10:24 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561954217/real 1561954217] req@ffff8f1621e07b00 x1636719604468016/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561954224 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 21:10:38 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1561954231/real 1561954231] req@ffff8f1c837be900 x1636719604789072/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1561954238 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 30 21:10:38 fir-md1-s1 kernel: Lustre: 24577:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 30 21:10:38 fir-md1-s1 kernel: LustreError: 24577:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.9@o2ib6) returned error from blocking AST (req@ffff8f1c837be900 x1636719604789072 status -107 rc -107), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f202add4a40/0x5d9ee62b06538b43 lrc: 4/0,0 mode: PR/PR res: [0x200029c10:0x348:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x4e1a1dca18a94de1 expref: 1589089 pid: 97642 timeout: 1069327 lvb_type: 0 Jun 30 21:10:38 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.9.9@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Jun 30 21:10:38 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 7s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f202add4a40/0x5d9ee62b06538b43 lrc: 3/0,0 mode: PR/PR res: [0x200029c10:0x348:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x4e1a1dca18a94de1 expref: 1589090 pid: 97642 timeout: 0 lvb_type: 0 Jun 30 21:10:38 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Jun 30 21:10:38 fir-md1-s1 kernel: LustreError: 20720:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f162d6d4e00 x1636719604960880/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 21:10:56 fir-md1-s1 kernel: Lustre: 97669:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f201812b000 x1634311306061824/t0(0) o36->a6b91a43-6f67-a7e7-0e97-a87e8033e0cf@10.8.9.10@o2ib6:1/0 lens 488/3152 e 0 to 0 dl 1561954261 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 21:11:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jun 30 21:11:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) Jun 30 21:11:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jun 30 21:11:27 fir-md1-s1 kernel: Lustre: 97664:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-29), not sending early reply req@ffff8f1b0cf08000 x1634311306064976/t0(0) o101->a6b91a43-6f67-a7e7-0e97-a87e8033e0cf@10.8.9.10@o2ib6:2/0 lens 480/568 e 0 to 0 dl 1561954292 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 21:11:31 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 5cad2422-3e98-66d4-e9e4-0ce15d870f56 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3bdae53c00, cur 1561954291 expire 1561954141 last 1561954064 Jun 30 21:11:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jun 30 21:11:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 4c6d21f6-3e09-6b98-bf50-a29faf23fa85 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f204a384000, cur 1561954299 expire 1561954149 last 1561954072 Jun 30 21:12:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) Jun 30 21:12:04 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 30 21:12:07 fir-md1-s1 kernel: LustreError: 21447:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1cbb0a8f00 x1636719607006880/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 21:12:08 fir-md1-s1 kernel: LustreError: 24577:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561954238, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1611a9e540/0x5d9ee62b0b964066 lrc: 3/0,1 mode: --/PW res: [0x200029c10:0x348:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 24577 timeout: 0 lvb_type: 0 Jun 30 21:12:32 fir-md1-s1 kernel: Lustre: 24578:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1efa717500 x1633733050680096/t0(0) o101->00a6bf4a-1a11-675b-07eb-2392e93c70c7@10.8.29.8@o2ib6:7/0 lens 376/1600 e 0 to 0 dl 1561954357 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 21:12:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jun 30 21:12:35 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jun 30 21:12:36 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f249f61e540/0x5d9ee62b05d6e8ef lrc: 3/0,0 mode: PR/PR res: [0x200029c2b:0x238:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x4e1a1dca17d13b39 expref: 979562 pid: 97642 timeout: 1069416 lvb_type: 0 Jun 30 21:12:36 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jun 30 21:13:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) Jun 30 21:13:37 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 30 21:13:38 fir-md1-s1 kernel: LustreError: 21447:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f240ca1bf00 x1636719608694528/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 21:13:47 fir-md1-s1 kernel: LustreError: 97643:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f2216ee2a00 x1636719608888272/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 21:13:47 fir-md1-s1 kernel: LustreError: 97643:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Jun 30 21:13:52 fir-md1-s1 kernel: LNet: Service thread pid 24577 was inactive for 200.68s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 30 21:13:52 fir-md1-s1 kernel: Pid: 24577, comm: mdt01_055 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 30 21:13:52 fir-md1-s1 kernel: Call Trace: Jun 30 21:13:52 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 30 21:13:52 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 30 21:13:52 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jun 30 21:13:52 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 30 21:13:52 fir-md1-s1 kernel: [] mdt_reint_object_lock+0x2c/0x60 [mdt] Jun 30 21:13:52 fir-md1-s1 kernel: [] mdt_reint_striped_lock+0x8c/0x510 [mdt] Jun 30 21:13:52 fir-md1-s1 kernel: [] mdt_reint_setattr+0x6c8/0x1340 [mdt] Jun 30 21:13:52 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jun 30 21:13:52 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jun 30 21:13:52 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jun 30 21:13:52 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 30 21:13:52 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 30 21:13:52 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 30 21:13:52 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 30 21:13:52 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 30 21:13:52 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 30 21:13:52 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561954432.24577 Jun 30 21:14:03 fir-md1-s1 kernel: Lustre: 22282:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1e32657800 x1636443528380064/t0(0) o101->7b7e9b9d-7d80-a5c4-07fd-dd92cbcbe2f0@10.8.29.6@o2ib6:8/0 lens 480/568 e 0 to 0 dl 1561954448 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 21:14:07 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f24eca28d80/0x5d9ee62b05f3f3c8 lrc: 3/0,0 mode: PR/PR res: [0x2000297f7:0x1ae:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x4e1a1dca17fff5bb expref: 778082 pid: 24577 timeout: 1069507 lvb_type: 0 Jun 30 21:14:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jun 30 21:14:08 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jun 30 21:15:08 fir-md1-s1 kernel: LustreError: 21447:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561954418, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1dc5285e80/0x5d9ee62b0c200537 lrc: 3/0,1 mode: --/PW res: [0x2000297f7:0x1ae:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21447 timeout: 0 lvb_type: 0 Jun 30 21:15:08 fir-md1-s1 kernel: LustreError: 21447:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jun 30 21:15:13 fir-md1-s1 kernel: LustreError: 97644:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561954423, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1c101572c0/0x5d9ee62b0c248f29 lrc: 3/0,1 mode: --/PW res: [0x200025b09:0x2436:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97644 timeout: 0 lvb_type: 0 Jun 30 21:15:13 fir-md1-s1 kernel: LustreError: 97644:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages Jun 30 21:15:43 fir-md1-s1 kernel: LustreError: 26254:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f21e3cda100 x1636719610292176/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 21:15:43 fir-md1-s1 kernel: LustreError: 26254:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jun 30 21:16:11 fir-md1-s1 kernel: Lustre: 20461:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-8), not sending early reply req@ffff8f2503ac3c00 x1634120669143008/t0(0) o101->b37c54be-7fed-724b-d760-c5bd71b2a4e0@10.8.29.5@o2ib6:16/0 lens 1776/3288 e 0 to 0 dl 1561954576 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 21:16:11 fir-md1-s1 kernel: Lustre: 20461:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages Jun 30 21:16:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) Jun 30 21:16:12 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jun 30 21:16:17 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f24e375b180/0x5d9ee62ac8ed1602 lrc: 3/0,0 mode: PR/PR res: [0x200029c6b:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x4e1a1dc9fab4b610 expref: 557097 pid: 21481 timeout: 1069637 lvb_type: 0 Jun 30 21:16:17 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages Jun 30 21:16:43 fir-md1-s1 kernel: LustreError: 22282:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f188462d700 x1636719610629984/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 21:16:43 fir-md1-s1 kernel: LustreError: 22282:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jun 30 21:16:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jun 30 21:16:43 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jun 30 21:16:59 fir-md1-s1 kernel: LNet: Service thread pid 21447 was inactive for 200.69s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 30 21:16:59 fir-md1-s1 kernel: Pid: 21447, comm: mdt01_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 30 21:16:59 fir-md1-s1 kernel: Call Trace: Jun 30 21:16:59 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 30 21:16:59 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 30 21:16:59 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jun 30 21:16:59 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 30 21:16:59 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Jun 30 21:16:59 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Jun 30 21:16:59 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Jun 30 21:16:59 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jun 30 21:16:59 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jun 30 21:16:59 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jun 30 21:16:59 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 30 21:16:59 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 30 21:16:59 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 30 21:16:59 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 30 21:16:59 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 30 21:16:59 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 30 21:16:59 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 30 21:16:59 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561954619.21447 Jun 30 21:17:02 fir-md1-s1 kernel: LNet: Service thread pid 24581 was inactive for 200.26s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 30 21:17:02 fir-md1-s1 kernel: Pid: 24581, comm: mdt01_059 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 30 21:17:02 fir-md1-s1 kernel: Call Trace: Jun 30 21:17:02 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 30 21:17:02 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 30 21:17:02 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 30 21:17:02 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561954622.24581 Jun 30 21:17:02 fir-md1-s1 kernel: Pid: 21434, comm: mdt01_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 30 21:17:02 fir-md1-s1 kernel: Call Trace: Jun 30 21:17:02 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_reint_object_lock+0x2c/0x60 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_reint_striped_lock+0x8c/0x510 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_reint_setattr+0x6c8/0x1340 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jun 30 21:17:02 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 30 21:17:02 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 30 21:17:02 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 30 21:17:02 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 30 21:17:04 fir-md1-s1 kernel: LNet: Service thread pid 97644 was inactive for 200.77s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 30 21:17:04 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Jun 30 21:17:04 fir-md1-s1 kernel: Pid: 97644, comm: mdt01_083 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 30 21:17:04 fir-md1-s1 kernel: Call Trace: Jun 30 21:17:04 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 30 21:17:04 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 30 21:17:04 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jun 30 21:17:04 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 30 21:17:04 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Jun 30 21:17:04 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Jun 30 21:17:04 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Jun 30 21:17:04 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jun 30 21:17:04 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jun 30 21:17:04 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jun 30 21:17:04 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 30 21:17:04 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 30 21:17:04 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 30 21:17:04 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 30 21:17:04 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 30 21:17:04 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 30 21:17:04 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 30 21:17:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561954624.97644 Jun 30 21:17:14 fir-md1-s1 kernel: LustreError: 26254:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561954543, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f16b93bde80/0x5d9ee62b0c9c9a8e lrc: 3/0,1 mode: --/CW res: [0x200011529:0x9f44:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 26254 timeout: 0 lvb_type: 0 Jun 30 21:17:14 fir-md1-s1 kernel: LustreError: 26254:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages Jun 30 21:17:31 fir-md1-s1 kernel: LNet: Service thread pid 21447 completed after 233.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 30 21:17:53 fir-md1-s1 kernel: LNet: Service thread pid 24577 completed after 441.62s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 30 21:17:57 fir-md1-s1 kernel: LustreError: 97671:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1e4de1e000 x1636719611004416/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jun 30 21:17:57 fir-md1-s1 kernel: LustreError: 97671:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jun 30 21:18:34 fir-md1-s1 kernel: LustreError: 97643:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561954624, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1f8fbe9680/0x5d9ee62b0cc2e4f6 lrc: 3/0,1 mode: --/CW res: [0x200025ed5:0x35cc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97643 timeout: 0 lvb_type: 0 Jun 30 21:18:34 fir-md1-s1 kernel: LustreError: 97643:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jun 30 21:19:04 fir-md1-s1 kernel: LNet: Service thread pid 26254 was inactive for 200.42s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 30 21:19:04 fir-md1-s1 kernel: Pid: 26254, comm: mdt01_067 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 30 21:19:04 fir-md1-s1 kernel: Call Trace: Jun 30 21:19:04 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 30 21:19:04 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 30 21:19:04 fir-md1-s1 kernel: [] mdt_object_local_lock+0x438/0xb20 [mdt] Jun 30 21:19:04 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 30 21:19:04 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Jun 30 21:19:04 fir-md1-s1 kernel: [] mdt_reint_open+0xc58/0x28b0 [mdt] Jun 30 21:19:04 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jun 30 21:19:04 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jun 30 21:19:04 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x350 [mdt] Jun 30 21:19:04 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jun 30 21:19:04 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jun 30 21:19:04 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jun 30 21:19:04 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 30 21:19:04 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 30 21:19:04 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 30 21:19:04 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 30 21:19:04 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 30 21:19:04 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 30 21:19:04 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 30 21:19:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561954744.26254 Jun 30 21:19:27 fir-md1-s1 kernel: LustreError: 97671:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1561954677, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f25237fd100/0x5d9ee62b0ce98cfe lrc: 3/0,1 mode: --/CW res: [0x20000fd6b:0x1fd91:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97671 timeout: 0 lvb_type: 0 Jun 30 21:20:25 fir-md1-s1 kernel: LNet: Service thread pid 97643 was inactive for 200.51s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 30 21:20:25 fir-md1-s1 kernel: Pid: 97643, comm: mdt01_082 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jun 30 21:20:25 fir-md1-s1 kernel: Call Trace: Jun 30 21:20:25 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jun 30 21:20:25 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jun 30 21:20:25 fir-md1-s1 kernel: [] mdt_object_local_lock+0x438/0xb20 [mdt] Jun 30 21:20:25 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jun 30 21:20:25 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Jun 30 21:20:25 fir-md1-s1 kernel: [] mdt_reint_open+0xc58/0x28b0 [mdt] Jun 30 21:20:25 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jun 30 21:20:25 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jun 30 21:20:25 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x350 [mdt] Jun 30 21:20:25 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jun 30 21:20:25 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jun 30 21:20:25 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jun 30 21:20:25 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 30 21:20:25 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jun 30 21:20:25 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jun 30 21:20:25 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jun 30 21:20:25 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jun 30 21:20:25 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jun 30 21:20:25 fir-md1-s1 kernel: [] 0xffffffffffffffff Jun 30 21:20:25 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1561954825.97643 Jun 30 21:21:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0af4f40a-317e-88ce-7d9c-c4839b78e5a4 (at 10.8.29.6@o2ib6) Jun 30 21:21:24 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jun 30 21:21:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 7b7e9b9d-7d80-a5c4-07fd-dd92cbcbe2f0 (at 10.8.29.6@o2ib6) reconnecting Jun 30 21:21:55 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jun 30 21:21:55 fir-md1-s1 kernel: LNet: Service thread pid 24581 completed after 493.65s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 30 21:22:14 fir-md1-s1 kernel: LNet: Service thread pid 97643 completed after 309.47s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 30 21:22:23 fir-md1-s1 kernel: Lustre: 97644:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-8), not sending early reply req@ffff8f1ed6691800 x1634120669206288/t0(0) o101->b37c54be-7fed-724b-d760-c5bd71b2a4e0@10.8.29.5@o2ib6:28/0 lens 576/3264 e 0 to 0 dl 1561954948 ref 2 fl Interpret:/0/0 rc 0/0 Jun 30 21:22:23 fir-md1-s1 kernel: Lustre: 97644:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jun 30 21:22:33 fir-md1-s1 kernel: LNet: Service thread pid 21434 completed after 531.82s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 30 21:22:33 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Jun 30 21:23:21 fir-md1-s1 kernel: LNet: Service thread pid 26254 completed after 457.82s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 30 22:04:17 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jun 30 22:04:17 fir-md1-s1 kernel: LustreError: 65760:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 22:04:57 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jun 30 22:04:57 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 30 previous similar messages Jun 30 22:08:29 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 22:08:29 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 56 previous similar messages Jun 30 22:36:00 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 22:38:46 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 22:38:46 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 23:06:25 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 23:09:11 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 23:09:11 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 23:36:49 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 23:39:41 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jun 30 23:39:41 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jun 30 23:55:59 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 01 00:40:58 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 01 00:41:08 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 01 00:41:08 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 00:41:33 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 01 00:41:33 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 00:41:48 fir-md1-s1 kernel: LustreError: 21289:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 01 00:41:48 fir-md1-s1 kernel: LustreError: 21289:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 00:42:13 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 36864 GRANT, real grant 0 Jul 01 00:42:13 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 00:42:43 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 36864 GRANT, real grant 0 Jul 01 00:42:43 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 00:43:33 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 36864 GRANT, real grant 0 Jul 01 00:43:33 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 01 00:44:08 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 40960 GRANT, real grant 0 Jul 01 00:44:08 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 01 00:45:14 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 36864 GRANT, real grant 0 Jul 01 00:45:14 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 01 00:47:42 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 01 00:47:42 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 01 00:52:00 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 36864 GRANT, real grant 0 Jul 01 00:52:00 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jul 01 02:43:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client f37c3da1-0e56-86e1-dca2-c29b3ae80868 (at 10.9.112.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148b08d800, cur 1561974224 expire 1561974074 last 1561973997 Jul 01 03:38:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 5b03b3b6-9c4d-bfc6-6338-bc8f69dac2d7 (at 10.9.105.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2521bca800, cur 1561977502 expire 1561977352 last 1561977275 Jul 01 03:38:22 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 03:42:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client fa50e0be-bf58-6b43-a8b3-a284779ef524 (at 10.8.13.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24ee74e400, cur 1561977752 expire 1561977602 last 1561977525 Jul 01 03:42:32 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 03:43:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 81206100-f2ef-5b10-2ad6-4678a9c95a5d (at 10.8.11.16@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff8f2522712000, cur 1561977828 expire 1561977678 last 1561977657 Jul 01 03:43:48 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 01 04:46:49 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 0f937866-81ed-1fa0-6cab-7aae3323fc7a (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f453a3d2c00, cur 1561981609 expire 1561981459 last 1561981382 Jul 01 04:46:49 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 04:50:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 0f937866-81ed-1fa0-6cab-7aae3323fc7a (at 10.8.11.20@o2ib6) Jul 01 04:50:53 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 01 06:47:39 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 06:47:39 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 01 08:28:09 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 08:28:09 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 01 09:08:23 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 09:08:23 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 09:38:51 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 09:45:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6b8f3c35-570b-9d9c-7deb-30e6f23700dc (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f190e7cf000, cur 1561999549 expire 1561999399 last 1561999322 Jul 01 09:45:49 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 09:45:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ffa27290-6cf4-9b77-ab2a-7df1aa693fad (at 10.8.21.21@o2ib6) Jul 01 09:45:52 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:08:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client a336b8b2-1d90-8ceb-26db-5f246ea4b144 (at 10.8.28.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2522d64000, cur 1562000931 expire 1562000781 last 1562000704 Jul 01 10:08:51 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:09:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 749699ee-a0f2-6ab2-f022-71007184e2c9 (at 10.8.8.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1489d17800, cur 1562000951 expire 1562000801 last 1562000724 Jul 01 10:09:11 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 01 10:23:12 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 10:23:12 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 10:24:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6917e80a-ef67-f4c1-8e7b-9c14a42b1479 (at 10.9.106.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f251fc94000, cur 1562001882 expire 1562001732 last 1562001655 Jul 01 10:24:42 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 01 10:25:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 003a2101-a39b-a323-1cef-2a0a958a28de (at 10.9.106.22@o2ib4) in 226 seconds. I think it's dead, and I am evicting it. exp ffff8f4523a3f400, cur 1562001958 expire 1562001808 last 1562001732 Jul 01 10:25:58 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:30:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.113.7@o2ib4) Jul 01 10:30:55 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:31:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1874c4f4-ebcb-1671-9e3e-6934890254c1 (at 10.9.115.6@o2ib4) Jul 01 10:31:06 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:33:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c0904b9-a746-baa3-6518-92bf7219376b (at 10.9.108.21@o2ib4) Jul 01 10:33:59 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:35:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jul 01 10:35:21 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:36:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 01549a7c-4c64-1571-057a-1e929c6f1684 (at 10.8.27.5@o2ib6) Jul 01 10:36:05 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:36:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to a07a243c-4ef8-8b68-a74f-ac2c8e98de57 (at 10.8.23.36@o2ib6) Jul 01 10:36:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:37:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to dee4f99a-2654-25ae-e6ec-cb4bc3f136c5 (at 10.8.28.6@o2ib6) Jul 01 10:37:07 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 01 10:38:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c579ffa9-959a-5f2e-006d-9d0dfdb5fa5a (at 10.8.17.26@o2ib6) Jul 01 10:38:23 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 01 10:41:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d974139a-4a0e-a5af-6c7c-02323898e17e (at 10.8.13.7@o2ib6) Jul 01 10:41:02 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 01 10:48:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1cdcf44c-092e-67dd-29a2-3cb7e9bc7e29 (at 10.8.15.6@o2ib6) Jul 01 10:48:49 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 10:54:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ac3cabc8-c0a0-bc39-c3a2-f19e3898f019 (at 10.9.107.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d2e2c1400, cur 1562003641 expire 1562003491 last 1562003414 Jul 01 10:54:01 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 01 11:17:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1cdcf44c-092e-67dd-29a2-3cb7e9bc7e29 (at 10.8.15.6@o2ib6) Jul 01 11:17:57 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 01 11:18:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.15.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 01 11:21:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 707ff730-8051-57ea-574b-4ed1b41d91e5 (at 10.9.106.22@o2ib4) Jul 01 11:21:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 11:24:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 98a67850-1b7c-ef40-1816-b3372d04b91a (at 10.9.104.26@o2ib4) Jul 01 11:24:16 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 11:45:57 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 12:33:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 815d7676-5c34-1cc9-c5dd-bad0fb6e70bb (at 10.8.14.8@o2ib6) Jul 01 12:33:27 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 01 13:59:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1cdcf44c-092e-67dd-29a2-3cb7e9bc7e29 (at 10.8.15.6@o2ib6) Jul 01 13:59:22 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 14:05:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 09178838-ce52-4043-1e0e-21a0c9717f63 (at 10.9.106.52@o2ib4) Jul 01 14:05:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 14:23:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to f2595515-8d55-d4e7-ea74-00e6bd9e71d3 (at 10.9.112.9@o2ib4) Jul 01 14:23:30 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 14:30:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c2a05b4-f659-9028-b43b-812cba74e3fc (at 10.9.106.70@o2ib4) Jul 01 14:30:19 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 14:59:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 54e0fe0b-05a8-2283-e1e6-c0953941c584 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3a0cfc5000, cur 1562018398 expire 1562018248 last 1562018171 Jul 01 14:59:58 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 01 15:00:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 54e0fe0b-05a8-2283-e1e6-c0953941c584 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0517995000, cur 1562018401 expire 1562018251 last 1562018174 Jul 01 15:00:01 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 01 15:00:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d8cc7b58-ee01-5501-ca65-c659f4724147 (at 10.9.106.54@o2ib4) Jul 01 15:00:44 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 15:11:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 3dc3e1e7-01f9-9795-87bb-84df780116dc (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148a475400, cur 1562019065 expire 1562018915 last 1562018838 Jul 01 15:11:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 3dc3e1e7-01f9-9795-87bb-84df780116dc (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2521889800, cur 1562019067 expire 1562018917 last 1562018840 Jul 01 15:11:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 01 15:14:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1af78b07-b135-5fa3-6c26-790cdde827a0 (at 10.9.113.5@o2ib4) Jul 01 15:14:44 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 15:15:27 fir-md1-s1 kernel: Lustre: 23582:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562019320/real 1562019320] req@ffff8f087f2edd00 x1636721314172816/t0(0) o104->fir-MDT0002@10.8.15.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562019327 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 01 15:15:34 fir-md1-s1 kernel: Lustre: 23582:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562019327/real 1562019327] req@ffff8f087f2edd00 x1636721314172816/t0(0) o104->fir-MDT0002@10.8.15.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562019334 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 01 15:15:35 fir-md1-s1 kernel: Lustre: 21145:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1476de5100 x1636462019526192/t0(0) o101->2d9198da-101c-d19d-2b4a-c0e67a82ee58@10.9.115.13@o2ib4:10/0 lens 1784/3288 e 1 to 0 dl 1562019340 ref 2 fl Interpret:/0/0 rc 0/0 Jul 01 15:15:41 fir-md1-s1 kernel: Lustre: 23582:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562019334/real 1562019334] req@ffff8f087f2edd00 x1636721314172816/t0(0) o104->fir-MDT0002@10.8.15.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562019341 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 01 15:15:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 2d9198da-101c-d19d-2b4a-c0e67a82ee58 (at 10.9.115.13@o2ib4) reconnecting Jul 01 15:15:41 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 01 15:15:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 28a67c6c-68a9-127c-f2e6-9416760ecb77 (at 10.9.115.13@o2ib4) Jul 01 15:15:41 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 15:15:55 fir-md1-s1 kernel: Lustre: 23582:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562019348/real 1562019348] req@ffff8f087f2edd00 x1636721314172816/t0(0) o104->fir-MDT0002@10.8.15.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562019355 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 01 15:15:55 fir-md1-s1 kernel: Lustre: 23582:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 01 15:15:55 fir-md1-s1 kernel: LustreError: 23582:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.15.4@o2ib6) failed to reply to blocking AST (req@ffff8f087f2edd00 x1636721314172816 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f22b045e0c0/0x5d9ee62c43f856dc lrc: 4/0,0 mode: PR/PR res: [0x2c002c313:0x8a9e:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.15.4@o2ib6 remote: 0x987fde9b9559811d expref: 1079 pid: 97651 timeout: 1134437 lvb_type: 0 Jul 01 15:15:55 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.15.4@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 01 15:15:55 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.15.4@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f22b045e0c0/0x5d9ee62c43f856dc lrc: 3/0,0 mode: PR/PR res: [0x2c002c313:0x8a9e:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.15.4@o2ib6 remote: 0x987fde9b9559811d expref: 1080 pid: 97651 timeout: 0 lvb_type: 0 Jul 01 15:15:55 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Jul 01 15:18:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client e67c9bb1-bbe7-aaeb-bb52-bf9dda890aef (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25216c1400, cur 1562019523 expire 1562019373 last 1562019296 Jul 01 15:36:25 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 15:36:25 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 15:48:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 15612231-22fb-a7bb-cd40-727fbf0eb380 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3d50240800, cur 1562021306 expire 1562021156 last 1562021079 Jul 01 15:48:26 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 01 15:48:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ffa27290-6cf4-9b77-ab2a-7df1aa693fad (at 10.8.21.21@o2ib6) Jul 01 15:52:47 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 15:52:48 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 15:52:48 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 01 15:52:49 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 15:52:49 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 01 15:52:51 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 15:52:51 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 01 15:53:18 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 15:53:18 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 01 15:53:27 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 15:53:27 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 43 previous similar messages Jul 01 15:53:54 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 15:53:54 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 01 15:54:39 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 15:54:39 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 31 previous similar messages Jul 01 15:56:26 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 15:56:26 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jul 01 15:59:42 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 15:59:42 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7 previous similar messages Jul 01 16:04:30 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 16:04:30 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 01 16:14:00 fir-md1-s1 kernel: LustreError: 69438:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 16:14:00 fir-md1-s1 kernel: LustreError: 69438:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 109 previous similar messages Jul 01 16:30:07 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 16:30:07 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 38 previous similar messages Jul 01 16:41:28 fir-md1-s1 kernel: LustreError: 46579:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 16:41:28 fir-md1-s1 kernel: LustreError: 46579:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 117 previous similar messages Jul 01 16:52:33 fir-md1-s1 kernel: LustreError: 69435:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 16:52:33 fir-md1-s1 kernel: LustreError: 69435:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 11 previous similar messages Jul 01 17:02:37 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 17:02:37 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 111 previous similar messages Jul 01 17:25:39 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 17:25:39 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 01 17:27:07 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 17:27:07 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 97 previous similar messages Jul 01 17:29:48 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 17:29:48 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 01 17:39:39 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 17:39:39 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 34 previous similar messages Jul 01 17:50:03 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 17:50:03 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 105 previous similar messages Jul 01 18:43:14 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 18:43:14 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 29 previous similar messages Jul 01 18:44:45 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 18:44:45 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 70 previous similar messages Jul 01 18:48:55 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 18:48:55 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 18 previous similar messages Jul 01 19:21:03 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 01 19:21:03 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 01 19:32:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 815d7676-5c34-1cc9-c5dd-bad0fb6e70bb (at 10.8.14.8@o2ib6) Jul 01 19:32:55 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 21:50:07 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 22:48:26 fir-md1-s1 kernel: LustreError: 69435:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 01 22:50:44 fir-md1-s1 kernel: Lustre: 69435:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f217734fc50 x1637891143276192/t0(0) o4->9a853c02-c745-a56d-0dbc-5a9440eb652f@10.8.15.6@o2ib6:19/0 lens 488/448 e 1 to 0 dl 1562046649 ref 2 fl Interpret:/0/0 rc 0/0 Jul 01 22:50:49 fir-md1-s1 kernel: LustreError: 21708:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f217734c850 x1637891143276256/t0(0) o4->9a853c02-c745-a56d-0dbc-5a9440eb652f@10.8.15.6@o2ib6:19/0 lens 488/448 e 1 to 0 dl 1562046649 ref 1 fl Interpret:/0/0 rc 0/0 Jul 01 22:50:49 fir-md1-s1 kernel: LustreError: 46562:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f21c9b4d450 x1637891143276208/t0(0) o4->9a853c02-c745-a56d-0dbc-5a9440eb652f@10.8.15.6@o2ib6:19/0 lens 488/448 e 1 to 0 dl 1562046649 ref 1 fl Interpret:/0/0 rc 0/0 Jul 01 22:50:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO write error with 9a853c02-c745-a56d-0dbc-5a9440eb652f (at 10.8.15.6@o2ib6), client will retry: rc = -110 Jul 01 22:50:49 fir-md1-s1 kernel: LustreError: 21708:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message Jul 01 22:53:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9a853c02-c745-a56d-0dbc-5a9440eb652f (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2515b47000, cur 1562046834 expire 1562046684 last 1562046607 Jul 01 22:53:54 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 01 22:54:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9a853c02-c745-a56d-0dbc-5a9440eb652f (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f23ae905400, cur 1562046856 expire 1562046706 last 1562046629 Jul 01 22:54:16 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 01 23:23:42 fir-md1-s1 kernel: LustreError: 69438:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 00:22:23 fir-md1-s1 kernel: LNetError: 20193:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jul 02 00:22:23 fir-md1-s1 kernel: LNetError: 20193:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 16 previous similar messages Jul 02 00:24:05 fir-md1-s1 kernel: LNetError: 20184:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jul 02 00:24:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 815d7676-5c34-1cc9-c5dd-bad0fb6e70bb (at 10.8.14.8@o2ib6) Jul 02 00:24:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 01:31:01 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 03:15:56 fir-md1-s1 kernel: LustreError: 46550:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 03:17:50 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:23:08 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:24:27 fir-md1-s1 kernel: LustreError: 46550:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:24:27 fir-md1-s1 kernel: LustreError: 46550:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 02 06:24:57 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:25:37 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:26:17 fir-md1-s1 kernel: LustreError: 46550:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:26:53 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:27:10 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:27:46 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:28:59 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:28:59 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 02 06:31:30 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:31:30 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 02 06:35:48 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:35:48 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7 previous similar messages Jul 02 06:44:51 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:44:51 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 02 06:55:02 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 06:55:02 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 02 07:05:04 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 07:05:04 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jul 02 07:15:25 fir-md1-s1 kernel: LustreError: 46550:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 07:15:25 fir-md1-s1 kernel: LustreError: 46550:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jul 02 07:25:48 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 07:25:48 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 27 previous similar messages Jul 02 07:27:05 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client cf80c3b1-3a35-aa95-401d-bdf5eda594e5 (at 10.9.105.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14758ea800, cur 1562077625 expire 1562077475 last 1562077398 Jul 02 07:27:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 55b02c38-d9ce-c2f6-066c-e168569494ff (at 10.9.105.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4505750400, cur 1562077631 expire 1562077481 last 1562077404 Jul 02 07:27:11 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 02 07:35:56 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 07:35:56 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 02 07:46:33 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 07:46:33 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 11 previous similar messages Jul 02 08:04:54 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 08:04:54 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 02 08:18:43 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 08:18:43 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 02 09:09:34 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 09:09:34 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 02 09:10:49 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 09:10:49 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 02 09:13:42 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 09:13:42 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jul 02 09:19:15 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 09:19:15 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 7 previous similar messages Jul 02 09:30:15 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 09:30:15 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jul 02 09:40:23 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 09:40:23 fir-md1-s1 kernel: LustreError: 21736:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 02 09:51:05 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 09:51:05 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 02 10:01:05 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 10:01:05 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 24 previous similar messages Jul 02 10:11:08 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 10:11:08 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 02 10:21:51 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 10:21:51 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 02 10:32:21 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 10:32:21 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 02 10:42:22 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 10:42:22 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22 previous similar messages Jul 02 10:52:31 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 10:52:31 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 21 previous similar messages Jul 02 11:02:34 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 11:02:34 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 21 previous similar messages Jul 02 11:13:35 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 11:13:35 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 18 previous similar messages Jul 02 11:23:42 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 11:23:42 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 02 11:33:48 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 11:33:48 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jul 02 11:43:58 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 11:43:58 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 34 previous similar messages Jul 02 11:53:59 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 11:53:59 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 37 previous similar messages Jul 02 12:04:24 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 12:04:24 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jul 02 12:14:45 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 12:14:45 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jul 02 12:24:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 49d65b51-f641-7136-faca-91f4ee67f9ec (at 10.9.0.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fe518000, cur 1562095462 expire 1562095312 last 1562095235 Jul 02 12:25:08 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 12:25:08 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 02 12:57:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b6020dc6-5ae0-1fda-6229-432d9300dcb9 (at 10.9.0.61@o2ib4) Jul 02 12:57:59 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 13:07:41 fir-md1-s1 kernel: Lustre: MGS: Connection restored to e4594a87-2fe5-1bf8-dbe3-26a702178742 (at 10.8.0.67@o2ib6) Jul 02 13:07:41 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 13:30:12 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 13:30:12 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 10 previous similar messages Jul 02 13:38:41 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 14:04:51 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 14:06:45 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 16:16:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 088aec52-9508-3401-0290-3c12a91037c4 (at 10.9.106.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25227a8400, cur 1562109386 expire 1562109236 last 1562109159 Jul 02 16:16:26 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 02 16:43:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 18baa8eb-3796-4c59-4335-f1e0f1008b8c (at 10.9.112.8@o2ib4) Jul 02 16:43:15 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 16:43:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to bfd7f797-6fd1-93d6-b01a-220fa07218f9 (at 10.9.112.10@o2ib4) Jul 02 16:43:45 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 16:44:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1cdcf44c-092e-67dd-29a2-3cb7e9bc7e29 (at 10.8.15.6@o2ib6) Jul 02 16:44:00 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 16:44:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 29e229ef-0b7d-e0ce-48dd-1c614dad7928 (at 10.9.112.15@o2ib4) Jul 02 16:44:19 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 16:45:50 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 2eee0836-3c27-6ecc-2655-fed0ce55b4ff (at 10.8.15.4@o2ib6) Jul 02 16:45:50 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 16:50:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to cf80c3b1-3a35-aa95-401d-bdf5eda594e5 (at 10.9.105.33@o2ib4) Jul 02 16:50:24 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 16:50:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7c3f56db-f273-5d44-6d2d-7a51f76d6b18 (at 10.8.10.25@o2ib6) Jul 02 16:50:41 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 02 16:51:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5eebc5e0-8890-b45f-8a55-b17e54a4b047 (at 10.9.106.8@o2ib4) Jul 02 16:51:24 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jul 02 17:16:44 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 17:23:12 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 17:29:11 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 17:48:44 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 02 17:48:54 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 17:48:59 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 17:48:59 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jul 02 17:49:09 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 17:49:09 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 02 17:49:31 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 02 17:49:31 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 45 previous similar messages Jul 02 17:50:31 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 17:50:31 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 02 17:52:40 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 02 17:52:40 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 02 17:55:52 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 17:55:52 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 02 18:03:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c0904b9-a746-baa3-6518-92bf7219376b (at 10.9.108.21@o2ib4) Jul 02 18:03:52 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 02 18:13:56 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 02 18:14:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 09178838-ce52-4043-1e0e-21a0c9717f63 (at 10.9.106.52@o2ib4) Jul 02 18:14:16 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 18:15:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bb86db3d-e55d-5db8-5c35-0541a49637df (at 10.9.108.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f44fed62400, cur 1562116538 expire 1562116388 last 1562116311 Jul 02 18:15:38 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 02 18:16:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c0904b9-a746-baa3-6518-92bf7219376b (at 10.9.108.21@o2ib4) Jul 02 18:16:34 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 18:17:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to f49314e4-fa04-90d7-0408-2e73086197cd (at 10.9.106.6@o2ib4) Jul 02 18:17:54 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 18:18:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 749e811b-d2e8-801c-4ade-84f4076c00ba (at 10.9.106.59@o2ib4) Jul 02 18:18:53 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 18:19:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c2a05b4-f659-9028-b43b-812cba74e3fc (at 10.9.106.70@o2ib4) Jul 02 18:19:17 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 18:19:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 80fdd8bf-960f-e808-91e0-c54ca3723917 (at 10.9.106.10@o2ib4) Jul 02 18:19:44 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 02 18:35:13 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 02 19:58:54 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 6d83953a-c249-6e7c-b76b-0ad244494f27 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3268e39400, cur 1562122734 expire 1562122584 last 1562122507 Jul 02 19:58:54 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 02 20:12:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 29e229ef-0b7d-e0ce-48dd-1c614dad7928 (at 10.9.112.15@o2ib4) Jul 02 20:12:58 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 03 00:59:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 97fc9649-24c2-fb40-27c9-532fdd9ea1ac (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d76f71000, cur 1562140744 expire 1562140594 last 1562140517 Jul 03 00:59:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 01:00:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c6748fa-faf9-dbf4-7576-e7e488da698d (at 10.8.11.9@o2ib6) Jul 03 01:00:21 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 01:20:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 1d5c7f14-25b8-ffc5-6646-b0756a504223 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f203b3e0c00, cur 1562142002 expire 1562141852 last 1562141775 Jul 03 01:20:02 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 01:21:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c6748fa-faf9-dbf4-7576-e7e488da698d (at 10.8.11.9@o2ib6) Jul 03 01:21:15 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 02:18:01 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 02:40:18 fir-md1-s1 kernel: LustreError: 69438:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 02:40:18 fir-md1-s1 kernel: LustreError: 69438:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 03 08:35:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 90d881d2-bbfa-565d-91e5-ddef873ff667 (at 10.9.105.48@o2ib4) Jul 03 08:35:23 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 10:03:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 7ae46df0-95ff-edbe-35f2-1ea841efe69a (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3505e30800, cur 1562173383 expire 1562173233 last 1562173156 Jul 03 10:03:03 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 10:03:09 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 22280de3-e127-2943-2417-f27756433740 (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2501b05000, cur 1562173389 expire 1562173239 last 1562173162 Jul 03 10:03:09 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 10:41:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 22280de3-e127-2943-2417-f27756433740 (at 10.9.0.81@o2ib4) Jul 03 10:41:43 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 10:43:36 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 10:45:09 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 03 10:45:09 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 03 11:08:13 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562177286/real 1562177286] req@ffff8f40f3e09500 x1636723353746768/t0(0) o106->fir-MDT0002@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562177293 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 11:08:20 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562177293/real 1562177293] req@ffff8f40f3e09500 x1636723353746768/t0(0) o106->fir-MDT0002@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562177300 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 11:08:21 fir-md1-s1 kernel: Lustre: 10309:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3cf6e23300 x1637978060226256/t0(0) o101->96f77fe0-d0c2-629d-bb62-dcf685e7e47d@10.9.0.61@o2ib4:26/0 lens 480/568 e 1 to 0 dl 1562177306 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 11:08:21 fir-md1-s1 kernel: Lustre: 10309:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 03 11:08:27 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562177300/real 1562177300] req@ffff8f40f3e09500 x1636723353746768/t0(0) o106->fir-MDT0002@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562177307 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 11:08:41 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562177314/real 1562177314] req@ffff8f40f3e09500 x1636723353746768/t0(0) o106->fir-MDT0002@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562177321 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 11:08:41 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 03 11:09:02 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562177335/real 1562177335] req@ffff8f40f3e09500 x1636723353746768/t0(0) o106->fir-MDT0002@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562177342 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 11:09:02 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 03 11:09:37 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562177370/real 1562177370] req@ffff8f40f3e09500 x1636723353746768/t0(0) o106->fir-MDT0002@10.8.1.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562177377 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 11:09:37 fir-md1-s1 kernel: Lustre: 23658:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 03 11:10:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 7c6bb3e9-46cc-b495-a385-90ef422e1faf (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f252340f000, cur 1562177403 expire 1562177253 last 1562177176 Jul 03 11:10:03 fir-md1-s1 kernel: Lustre: 23658:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:97s); client may timeout. req@ffff8f3cf6e23300 x1637978060226256/t0(0) o101->96f77fe0-d0c2-629d-bb62-dcf685e7e47d@10.9.0.61@o2ib4:26/0 lens 480/536 e 1 to 0 dl 1562177306 ref 1 fl Complete:/0/0 rc 301/301 Jul 03 11:10:03 fir-md1-s1 kernel: Lustre: 23658:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 25 previous similar messages Jul 03 11:12:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 26320709-561f-90ed-6684-fea46854b319 (at 10.8.1.29@o2ib6) Jul 03 11:12:28 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 11:20:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 903b60d0-f5a3-a51e-70de-07052e4bb832 (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148acc9000, cur 1562178041 expire 1562177891 last 1562177814 Jul 03 11:20:41 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 11:51:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 22280de3-e127-2943-2417-f27756433740 (at 10.9.0.81@o2ib4) Jul 03 11:51:02 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 12:02:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 2306b587-2a3d-1ec9-bc70-ef2318848cbd (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4504d25c00, cur 1562180544 expire 1562180394 last 1562180317 Jul 03 12:02:24 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 12:33:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 22280de3-e127-2943-2417-f27756433740 (at 10.9.0.81@o2ib4) Jul 03 12:33:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 12:37:35 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 176128 GRANT, real grant 0 Jul 03 12:38:09 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 147456 GRANT, real grant 0 Jul 03 12:38:37 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 147456 GRANT, real grant 0 Jul 03 12:38:44 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 151552 GRANT, real grant 0 Jul 03 12:39:14 fir-md1-s1 kernel: LustreError: 46560:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 147456 GRANT, real grant 0 Jul 03 13:38:55 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 98ff8e84-1e9a-d223-7706-0c3e5612efc7 (at 10.8.0.82@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f350da7f800, cur 1562186335 expire 1562186185 last 1562186108 Jul 03 13:38:55 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 13:39:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client ebb0ff39-b00e-6e1a-c25b-64754a77a1b9 (at 10.8.0.82@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2520743800, cur 1562186347 expire 1562186197 last 1562186120 Jul 03 13:55:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 336f7f9b-5dca-7bc0-f540-0bda4a5c5916 (at 10.9.106.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4518915000, cur 1562187328 expire 1562187178 last 1562187101 Jul 03 13:55:28 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 14:30:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 98ff8e84-1e9a-d223-7706-0c3e5612efc7 (at 10.8.0.82@o2ib6) Jul 03 14:30:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 14:33:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 3ec7adcc-54ba-9f81-9e8f-cc86aea17c81 (at 10.9.110.1@o2ib4) Jul 03 14:33:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 14:33:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c44cd238-8ca6-320e-a8ee-be68a5621e8e (at 10.9.110.2@o2ib4) Jul 03 14:33:59 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 03 15:26:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 58c0585b-a4ca-91b5-f3d9-6c740c9c6c69 (at 10.9.102.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2538047400, cur 1562192788 expire 1562192638 last 1562192561 Jul 03 15:26:28 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 03 15:33:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jul 03 15:33:48 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jul 03 15:34:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client dc8c296c-90b6-4272-e4d3-a5c935663898 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34daabd400, cur 1562193240 expire 1562193090 last 1562193013 Jul 03 15:34:00 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 03 15:35:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c3b139fe-a52f-1c45-3280-dbbeca16676d (at 10.8.23.14@o2ib6) in 217 seconds. I think it's dead, and I am evicting it. exp ffff8f1ec8a1a800, cur 1562193316 expire 1562193166 last 1562193099 Jul 03 15:35:16 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 15:35:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to acd26ab4-a020-fbc0-1a40-f0e7d759131f (at 10.8.23.14@o2ib6) Jul 03 15:35:26 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 15:42:58 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client e3fe702d-7407-7671-1296-c76bd9eb9ca1 (at 10.9.113.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14c7775c00, cur 1562193778 expire 1562193628 last 1562193551 Jul 03 15:42:58 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 15:45:11 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 15:45:14 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 15:45:34 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 15:45:34 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 10 previous similar messages Jul 03 15:45:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 761c131c-61da-c461-162b-cc2b93210f35 (at 10.9.102.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f453beaa000, cur 1562193947 expire 1562193797 last 1562193720 Jul 03 15:45:47 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 15:45:47 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 03 15:45:47 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jul 03 15:45:59 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 03 15:45:59 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 03 15:46:16 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 15:46:16 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 69 previous similar messages Jul 03 15:46:55 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 15:46:55 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 03 15:48:10 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 15:48:10 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 95 previous similar messages Jul 03 15:49:21 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562194154/real 1562194154] req@ffff8f404bf65700 x1636723390245824/t0(0) o104->fir-MDT0000@10.9.115.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562194161 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 15:49:21 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 03 15:49:29 fir-md1-s1 kernel: Lustre: 10586:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f392d7d7b00 x1634999679427376/t0(0) o36->b60a3bb6-bbe2-b613-59ad-fb772c2a43bc@10.9.107.65@o2ib4:4/0 lens 512/448 e 1 to 0 dl 1562194174 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 15:49:30 fir-md1-s1 kernel: Lustre: 23565:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f317a320000 x1631621340917520/t0(0) o101->c4a74d2b-de98-9a37-7ebb-5f19657dadd1@10.9.108.2@o2ib4:5/0 lens 584/3264 e 1 to 0 dl 1562194175 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 15:49:35 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562194168/real 1562194168] req@ffff8f404bf65700 x1636723390245824/t0(0) o104->fir-MDT0000@10.9.115.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562194175 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 15:49:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b60a3bb6-bbe2-b613-59ad-fb772c2a43bc (at 10.9.107.65@o2ib4) reconnecting Jul 03 15:49:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to f35d1ecc-fa81-1964-68b0-0ffaf770a8d3 (at 10.9.107.65@o2ib4) Jul 03 15:49:35 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 15:49:35 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 03 15:49:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to a0e5688c-3919-4790-9111-00c22859e271 (at 10.9.108.2@o2ib4) Jul 03 15:49:40 fir-md1-s1 kernel: Lustre: 20730:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1e39bb0900 x1631625914855264/t0(0) o101->b2acd6c0-c0f5-61d3-4a68-78d78ff1740e@10.8.27.13@o2ib6:15/0 lens 584/3264 e 1 to 0 dl 1562194185 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 15:49:40 fir-md1-s1 kernel: Lustre: 20730:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 03 15:49:42 fir-md1-s1 kernel: Lustre: 20730:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1682f9ec00 x1634620019363568/t0(0) o101->46725c7e-13ed-427c-fac8-b2b98cb851a6@10.8.17.12@o2ib6:17/0 lens 584/3264 e 1 to 0 dl 1562194187 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 15:49:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b2acd6c0-c0f5-61d3-4a68-78d78ff1740e (at 10.8.27.13@o2ib6) reconnecting Jul 03 15:49:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 15:49:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 57743c79-31a8-108e-7e60-aa89857aef81 (at 10.8.27.13@o2ib6) Jul 03 15:49:46 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 15:49:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to bfd7f797-6fd1-93d6-b01a-220fa07218f9 (at 10.9.112.10@o2ib4) Jul 03 15:49:52 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 15:49:56 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562194189/real 1562194189] req@ffff8f404bf65700 x1636723390245824/t0(0) o104->fir-MDT0000@10.9.115.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562194196 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 15:49:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to f35d1ecc-fa81-1964-68b0-0ffaf770a8d3 (at 10.9.107.65@o2ib4) Jul 03 15:49:56 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 15:49:56 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 03 15:50:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b2acd6c0-c0f5-61d3-4a68-78d78ff1740e (at 10.8.27.13@o2ib6) reconnecting Jul 03 15:50:07 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 03 15:50:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 57743c79-31a8-108e-7e60-aa89857aef81 (at 10.8.27.13@o2ib6) Jul 03 15:50:07 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 15:50:15 fir-md1-s1 kernel: Lustre: 22279:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/-6), not sending early reply req@ffff8f2530bab000 x1637395057955728/t0(0) o101->65c7cbb7-edd7-61f5-c144-1ffbb9efedd7@10.8.1.35@o2ib6:19/0 lens 584/3264 e 0 to 0 dl 1562194219 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 15:50:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 57743c79-31a8-108e-7e60-aa89857aef81 (at 10.8.27.13@o2ib6) Jul 03 15:50:29 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 03 15:50:29 fir-md1-s1 kernel: Lustre: 10561:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f41d502dd00 x1634125092485888/t0(0) o101->2eaf5a11-c409-36b3-5d68-7ef19d1bd3f9@10.9.107.68@o2ib4:4/0 lens 584/3264 e 1 to 0 dl 1562194234 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 15:50:31 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562194224/real 1562194224] req@ffff8f404bf65700 x1636723390245824/t0(0) o104->fir-MDT0000@10.9.115.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562194231 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 15:50:31 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 03 15:50:32 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 15:50:32 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 216 previous similar messages Jul 03 15:50:45 fir-md1-s1 kernel: LustreError: 10196:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562194155, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f0513e7d7c0/0x5d9ee62e682e9560 lrc: 3/1,0 mode: --/PR res: [0x200011cf2:0x1b49:0x0].0x0 bits 0x13/0x0 rrc: 14 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10196 timeout: 0 lvb_type: 0 Jul 03 15:50:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6e48472f-542d-f444-2879-49b8d614290d (at 10.9.108.8@o2ib4) reconnecting Jul 03 15:50:48 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 03 15:50:49 fir-md1-s1 kernel: Lustre: 22288:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1821f8c200 x1634917573629072/t0(0) o101->bb6c1ebe-228f-c2b0-845a-14ae6de0b327@10.8.27.21@o2ib6:24/0 lens 584/3264 e 0 to 0 dl 1562194254 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 15:50:49 fir-md1-s1 kernel: Lustre: 22288:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jul 03 15:50:55 fir-md1-s1 kernel: LustreError: 97648:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562194165, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2063019680/0x5d9ee62e68494616 lrc: 3/1,0 mode: --/PR res: [0x200011cf2:0x1b49:0x0].0x0 bits 0x13/0x0 rrc: 15 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97648 timeout: 0 lvb_type: 0 Jul 03 15:50:55 fir-md1-s1 kernel: LustreError: 97648:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 03 15:51:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b5a4ca60-bdc7-f60e-1d00-41d316e40dac (at 10.9.108.3@o2ib4) Jul 03 15:51:03 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 03 15:51:19 fir-md1-s1 kernel: LustreError: 20726:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562194189, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1f2fcbcc80/0x5d9ee62e68616328 lrc: 3/1,0 mode: --/PR res: [0x200011cf2:0x1b49:0x0].0x0 bits 0x13/0x0 rrc: 17 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20726 timeout: 0 lvb_type: 0 Jul 03 15:51:19 fir-md1-s1 kernel: LustreError: 20726:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 03 15:51:24 fir-md1-s1 kernel: Lustre: 20730:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1f39946900 x1634290379835008/t0(0) o101->9081d826-2f83-5b46-ff73-7e6473184838@10.8.17.25@o2ib6:29/0 lens 584/3264 e 0 to 0 dl 1562194289 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 15:51:24 fir-md1-s1 kernel: Lustre: 20730:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 03 15:51:41 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562194294/real 1562194294] req@ffff8f404bf65700 x1636723390245824/t0(0) o104->fir-MDT0000@10.9.115.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562194301 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 15:51:41 fir-md1-s1 kernel: Lustre: 10363:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 03 15:51:48 fir-md1-s1 kernel: LustreError: 10363:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.115.6@o2ib4) failed to reply to blocking AST (req@ffff8f404bf65700 x1636723390245824 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f1bacc38d80/0x5d9ee62e59b154c1 lrc: 4/0,0 mode: PR/PR res: [0x200011cf2:0x1b49:0x0].0x0 bits 0x13/0x0 rrc: 19 type: IBT flags: 0x60200400000020 nid: 10.9.115.6@o2ib4 remote: 0x6755c310f7297395 expref: 19 pid: 10332 timeout: 1309511 lvb_type: 0 Jul 03 15:51:48 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.9.115.6@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 03 15:51:48 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.9.115.6@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff8f1bacc38d80/0x5d9ee62e59b154c1 lrc: 3/0,0 mode: PR/PR res: [0x200011cf2:0x1b49:0x0].0x0 bits 0x13/0x0 rrc: 19 type: IBT flags: 0x60200400000020 nid: 10.9.115.6@o2ib4 remote: 0x6755c310f7297395 expref: 20 pid: 10332 timeout: 0 lvb_type: 0 Jul 03 15:51:49 fir-md1-s1 kernel: Lustre: 10333:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:1s); client may timeout. req@ffff8f3a4e70b600 x1635092894709312/t0(0) o101->40fe7f0a-1b2a-cef5-fe8d-06bb6237455c@10.9.108.5@o2ib4:18/0 lens 584/536 e 0 to 0 dl 1562194308 ref 1 fl Complete:/0/0 rc 0/0 Jul 03 15:52:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client a545c53d-fd13-75ed-6bde-35d1aaac7a2f (at 10.9.106.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f250d915400, cur 1562194321 expire 1562194171 last 1562194094 Jul 03 15:52:01 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 03 15:52:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to df993956-2257-9a73-35ef-341b2f75d156 (at 10.9.106.58@o2ib4) Jul 03 15:52:40 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 03 15:54:48 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 15:54:48 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 506 previous similar messages Jul 03 15:55:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 0986c76b-92a3-5eb5-0ecd-38e58fcb1758 (at 10.8.26.34@o2ib6) Jul 03 15:55:02 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 03 15:55:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client a0cea963-bad4-2a43-1e1e-2b16d5cc26b0 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f45355fa400, cur 1562194531 expire 1562194381 last 1562194304 Jul 03 15:55:31 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 03 15:58:15 fir-md1-s1 kernel: Lustre: 24576:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562194688/real 1562194688] req@ffff8f1612a6f500 x1636723394558688/t0(0) o104->fir-MDT0002@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562194695 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 15:58:15 fir-md1-s1 kernel: Lustre: 24576:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 03 15:59:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 221c24d0-0082-781d-4acc-41656456a74c (at 10.9.106.58@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f451de73400, cur 1562194765 expire 1562194615 last 1562194538 Jul 03 15:59:25 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 03 15:59:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 556216e1-e907-ce15-d71c-dcbb67e6c0d6 (at 10.8.1.1@o2ib6) Jul 03 15:59:28 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jul 03 16:00:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.7@o2ib6, removing former export from same NID Jul 03 16:00:30 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 16:00:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 4343a906-23d9-f729-b768-bcd0549ada0d (at 10.8.8.37@o2ib6) reconnecting Jul 03 16:00:31 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 03 16:03:24 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 16:03:24 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 634 previous similar messages Jul 03 16:10:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ad179e1c-410d-6164-932b-33dee7383182 (at 10.9.114.8@o2ib4) Jul 03 16:10:19 fir-md1-s1 kernel: Lustre: Skipped 158 previous similar messages Jul 03 16:13:32 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 16:13:32 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 736 previous similar messages Jul 03 16:19:53 fir-md1-s1 kernel: Lustre: 97672:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562195986/real 1562195986] req@ffff8f1c3bbc6900 x1636723398952656/t0(0) o104->fir-MDT0002@10.8.15.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562195993 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 16:19:53 fir-md1-s1 kernel: Lustre: 97672:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 03 16:20:01 fir-md1-s1 kernel: Lustre: 10559:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f43a8a71e00 x1638080845415248/t0(0) o101->cac1eba7-cdaa-957f-8735-d5169807717b@10.9.112.9@o2ib4:6/0 lens 1784/3288 e 1 to 0 dl 1562196006 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:20:01 fir-md1-s1 kernel: Lustre: 10559:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 03 16:20:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client cac1eba7-cdaa-957f-8735-d5169807717b (at 10.9.112.9@o2ib4) reconnecting Jul 03 16:20:07 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 03 16:20:14 fir-md1-s1 kernel: Lustre: 23682:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3f8baef500 x1636461767715776/t0(0) o101->cd3d0230-3738-e2d9-7e9f-2fd94c27579a@10.9.115.5@o2ib4:18/0 lens 1784/3288 e 1 to 0 dl 1562196018 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:20:14 fir-md1-s1 kernel: Lustre: 23682:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 03 16:20:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 9f9c20f1-d776-b800-1cdc-f625bb18ebc2 (at 10.9.115.5@o2ib4) Jul 03 16:20:19 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 03 16:20:26 fir-md1-s1 kernel: Lustre: 21667:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562196019/real 1562196019] req@ffff8f377bbb0c00 x1636723398983888/t0(0) o104->fir-MDT0002@10.8.15.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562196026 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 16:20:26 fir-md1-s1 kernel: Lustre: 21667:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 03 16:20:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client cac1eba7-cdaa-957f-8735-d5169807717b (at 10.9.112.9@o2ib4) reconnecting Jul 03 16:20:28 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 16:21:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client cac1eba7-cdaa-957f-8735-d5169807717b (at 10.9.112.9@o2ib4) reconnecting Jul 03 16:21:10 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 03 16:21:20 fir-md1-s1 kernel: Lustre: 50442:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2af4f2a400 x1638080879728048/t0(0) o101->b3e5d320-3d62-bccd-461a-ac941a8ebc1b@10.9.112.8@o2ib4:25/0 lens 1784/3288 e 1 to 0 dl 1562196085 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:21:31 fir-md1-s1 kernel: Lustre: 97664:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562196084/real 1562196084] req@ffff8f1592759500 x1636723399224848/t0(0) o104->fir-MDT0002@10.8.15.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562196091 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 16:21:31 fir-md1-s1 kernel: Lustre: 97664:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 59 previous similar messages Jul 03 16:22:06 fir-md1-s1 kernel: Lustre: 10143:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-8), not sending early reply req@ffff8f3263b90c00 x1638081177340544/t0(0) o101->8a102ad1-e9a6-7534-f996-e08c017dc5d4@10.9.113.6@o2ib4:11/0 lens 576/3264 e 0 to 0 dl 1562196131 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:22:06 fir-md1-s1 kernel: Lustre: 10143:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages Jul 03 16:22:20 fir-md1-s1 kernel: LustreError: 97646:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.15.4@o2ib6) failed to reply to blocking AST (req@ffff8f19ddfca700 x1636723398953968 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f18f9b4cc80/0x5d9ee62e657bbf94 lrc: 4/0,0 mode: PR/PR res: [0x2c002c33b:0x36cc:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.15.4@o2ib6 remote: 0x181a13f4089308aa expref: 72634 pid: 97639 timeout: 1311342 lvb_type: 0 Jul 03 16:22:20 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.15.4@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 03 16:22:20 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.15.4@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f161e62b180/0x5d9ee62e657bb3e0 lrc: 3/0,0 mode: PR/PR res: [0x2c002c33b:0x36cb:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.15.4@o2ib6 remote: 0x181a13f40893062d expref: 72635 pid: 21429 timeout: 0 lvb_type: 0 Jul 03 16:22:20 fir-md1-s1 kernel: LustreError: 97646:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Jul 03 16:22:33 fir-md1-s1 kernel: LustreError: 23632:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f0513e58300 x1636723399418960/t0(0) o104->fir-MDT0002@10.8.15.4@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 03 16:22:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client cac1eba7-cdaa-957f-8735-d5169807717b (at 10.9.112.9@o2ib4) reconnecting Jul 03 16:22:34 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 03 16:23:10 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 0da3bac3-60e3-7f8e-ab8a-bd9e331cf431 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f450119dc00, cur 1562196190 expire 1562196040 last 1562195963 Jul 03 16:23:10 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 03 16:23:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 92fd2bcd-71b5-44d8-7ea5-53f463aabbb9 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1e9db27c00, cur 1562196202 expire 1562196052 last 1562195975 Jul 03 16:23:35 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 16:23:35 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 688 previous similar messages Jul 03 16:32:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 26ab021a-adb7-b814-3d61-a4e6dec4651f (at 10.8.9.9@o2ib6) reconnecting Jul 03 16:32:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.9.9@o2ib6, removing former export from same NID Jul 03 16:32:31 fir-md1-s1 kernel: Lustre: Skipped 128 previous similar messages Jul 03 16:32:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to a9a5dee3-366f-e94e-5233-92c151efbd27 (at 10.8.9.9@o2ib6) Jul 03 16:32:31 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 03 16:33:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.9.9@o2ib6, removing former export from same NID Jul 03 16:33:51 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 16:33:51 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 807 previous similar messages Jul 03 16:34:33 fir-md1-s1 kernel: Lustre: 23634:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562196866/real 1562196866] req@ffff8f14ce1e4200 x1636723401226512/t0(0) o104->fir-MDT0002@10.9.112.14@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562196873 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 16:34:33 fir-md1-s1 kernel: Lustre: 23634:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 86 previous similar messages Jul 03 16:34:41 fir-md1-s1 kernel: Lustre: 23602:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3689a47800 x1638005242811616/t0(0) o101->8ef25a02-5cd5-8500-774d-d75ea76eaffd@10.9.112.15@o2ib4:16/0 lens 1784/3288 e 1 to 0 dl 1562196886 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:34:55 fir-md1-s1 kernel: Lustre: 23634:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562196887/real 1562196887] req@ffff8f14ce1e4200 x1636723401226512/t0(0) o104->fir-MDT0002@10.9.112.14@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562196894 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 16:34:55 fir-md1-s1 kernel: Lustre: 23634:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 03 16:35:02 fir-md1-s1 kernel: LustreError: 23634:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.112.14@o2ib4) failed to reply to blocking AST (req@ffff8f14ce1e4200 x1636723401226512 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f0cfcff8000/0x5d9ee62e67f9e63c lrc: 4/0,0 mode: PR/PR res: [0x2c002c33b:0x370a:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.112.14@o2ib4 remote: 0x4b3e857652935417 expref: 39771 pid: 23589 timeout: 1311984 lvb_type: 0 Jul 03 16:35:02 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.9.112.14@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 03 16:35:02 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 03 16:35:02 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 36s: evicting client at 10.9.112.14@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f0cfcff8000/0x5d9ee62e67f9e63c lrc: 3/0,0 mode: PR/PR res: [0x2c002c33b:0x370a:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.112.14@o2ib4 remote: 0x4b3e857652935417 expref: 39772 pid: 23589 timeout: 0 lvb_type: 0 Jul 03 16:36:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.9.9@o2ib6, removing former export from same NID Jul 03 16:36:13 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client e79f4448-e890-1954-0996-0a25890d8ee5 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14d2227000, cur 1562196973 expire 1562196823 last 1562196746 Jul 03 16:36:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 791a2ecd-fea5-54c9-c926-e06f2b6d4ac4 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4538d74800, cur 1562196981 expire 1562196831 last 1562196754 Jul 03 16:36:21 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 16:36:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 03 16:36:27 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 03 16:37:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.9.9@o2ib6, removing former export from same NID Jul 03 16:37:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 73304416-dad3-e9c2-af6e-d3b1ea37367d (at 10.9.114.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148a5ff400, cur 1562197035 expire 1562196885 last 1562196808 Jul 03 16:37:15 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 16:40:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ba9d5bce-9de0-28f1-af07-112093ff61ad (at 10.9.106.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25210f5400, cur 1562197207 expire 1562197057 last 1562196980 Jul 03 16:43:27 fir-md1-s1 kernel: Lustre: 97672:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562197400/real 1562197400] req@ffff8f1c12ae8c00 x1636723403222368/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562197407 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 16:43:27 fir-md1-s1 kernel: Lustre: 97672:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 03 16:43:45 fir-md1-s1 kernel: Lustre: 21483:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f201f396f00 x1633733151575392/t0(0) o101->00a6bf4a-1a11-675b-07eb-2392e93c70c7@10.8.29.8@o2ib6:20/0 lens 376/1600 e 0 to 0 dl 1562197430 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:43:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 00a6bf4a-1a11-675b-07eb-2392e93c70c7 (at 10.8.29.8@o2ib6) reconnecting Jul 03 16:43:51 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 03 16:43:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 220a94f1-3873-c0d2-13c3-2a8b3b58132e (at 10.8.29.8@o2ib6) Jul 03 16:43:51 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 03 16:44:06 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 147456 GRANT, real grant 0 Jul 03 16:44:06 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 709 previous similar messages Jul 03 16:44:33 fir-md1-s1 kernel: Lustre: 97661:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562197466/real 1562197466] req@ffff8f1e25ad3600 x1636723403347008/t0(0) o106->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562197473 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 16:44:33 fir-md1-s1 kernel: Lustre: 97661:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 03 16:44:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client dd15fe9e-fdc9-c67d-748d-ca571be05b29 (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1b17674400, cur 1562197475 expire 1562197325 last 1562197248 Jul 03 16:44:35 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 16:44:48 fir-md1-s1 kernel: LustreError: 97661:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1e25ad3600 x1636723403376704/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 03 16:45:05 fir-md1-s1 kernel: LustreError: 24586:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f222cec9200 x1636723403406320/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 03 16:45:16 fir-md1-s1 kernel: LustreError: 24585:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f250a7c3f00 x1636723403423344/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 03 16:45:41 fir-md1-s1 kernel: Lustre: 97643:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f243d812700 x1633733151608160/t0(0) o101->00a6bf4a-1a11-675b-07eb-2392e93c70c7@10.8.29.8@o2ib6:16/0 lens 480/568 e 0 to 0 dl 1562197546 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:45:59 fir-md1-s1 kernel: LustreError: 21460:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1de0677200 x1636723403515488/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 03 16:46:24 fir-md1-s1 kernel: Lustre: 97643:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1a8af5f200 x1633733151622352/t0(0) o101->00a6bf4a-1a11-675b-07eb-2392e93c70c7@10.8.29.8@o2ib6:29/0 lens 480/568 e 0 to 0 dl 1562197589 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:47:29 fir-md1-s1 kernel: LustreError: 21460:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562197559, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f20c67ede80/0x5d9ee62e83d4b33a lrc: 3/1,0 mode: --/PR res: [0x200029c11:0xfa:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21460 timeout: 0 lvb_type: 0 Jul 03 16:47:29 fir-md1-s1 kernel: LustreError: 21460:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages Jul 03 16:48:28 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.8.15.6@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f24ed68d100/0x5d9ee62e7fcd33d9 lrc: 3/0,0 mode: PW/PW res: [0x200029c11:0xfa:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.8.15.6@o2ib6 remote: 0xba301179ab01cef5 expref: 922849 pid: 21483 timeout: 1312768 lvb_type: 0 Jul 03 16:49:19 fir-md1-s1 kernel: LNet: Service thread pid 21460 was inactive for 200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 03 16:49:19 fir-md1-s1 kernel: Pid: 21460, comm: mdt01_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 03 16:49:19 fir-md1-s1 kernel: Call Trace: Jul 03 16:49:19 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 03 16:49:19 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 03 16:49:19 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 03 16:49:19 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 03 16:49:19 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Jul 03 16:49:19 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Jul 03 16:49:19 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Jul 03 16:49:19 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 03 16:49:19 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 03 16:49:19 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 03 16:49:19 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 03 16:49:19 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 03 16:49:19 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 03 16:49:19 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 03 16:49:19 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 03 16:49:19 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 03 16:49:19 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 03 16:49:19 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1562197759.21460 Jul 03 16:49:25 fir-md1-s1 kernel: LustreError: 97671:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1fdf429500 x1636723403988224/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 03 16:50:02 fir-md1-s1 kernel: Lustre: 20511:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-17), not sending early reply req@ffff8f1df53fe300 x1633733151651008/t0(0) o101->00a6bf4a-1a11-675b-07eb-2392e93c70c7@10.8.29.8@o2ib6:7/0 lens 376/1600 e 0 to 0 dl 1562197807 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 16:50:55 fir-md1-s1 kernel: LustreError: 97671:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562197765, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f16241e3180/0x5d9ee62e8502b2f9 lrc: 3/0,1 mode: --/EX res: [0x200029c2b:0x351:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97671 timeout: 0 lvb_type: 0 Jul 03 16:51:54 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.8.15.6@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f4492e88b40/0x5d9ee62e80adff65 lrc: 3/0,0 mode: PR/PR res: [0x200029c2b:0x34f:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.15.6@o2ib6 remote: 0xba301179ab44a739 expref: 576551 pid: 22287 timeout: 1312974 lvb_type: 0 Jul 03 16:52:45 fir-md1-s1 kernel: LNet: Service thread pid 97664 was inactive for 200.07s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 03 16:52:45 fir-md1-s1 kernel: Pid: 97664, comm: mdt01_103 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 03 16:52:45 fir-md1-s1 kernel: Call Trace: Jul 03 16:52:45 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 03 16:52:45 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 03 16:52:45 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 03 16:52:45 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 03 16:52:45 fir-md1-s1 kernel: [] mdt_layout_change+0x2a4/0x430 [mdt] Jul 03 16:52:45 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Jul 03 16:52:45 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 03 16:52:45 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 03 16:52:45 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 03 16:52:45 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 03 16:52:45 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 03 16:52:45 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 03 16:52:45 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 03 16:52:46 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 03 16:52:46 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 03 16:52:46 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 03 16:52:46 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1562197966.97664 Jul 03 16:54:07 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 16:54:07 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 742 previous similar messages Jul 03 16:54:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 00a6bf4a-1a11-675b-07eb-2392e93c70c7 (at 10.8.29.8@o2ib6) reconnecting Jul 03 16:54:16 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 03 16:54:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 220a94f1-3873-c0d2-13c3-2a8b3b58132e (at 10.8.29.8@o2ib6) Jul 03 16:54:16 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 03 16:56:23 fir-md1-s1 kernel: LNet: Service thread pid 21460 completed after 624.39s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jul 03 16:57:17 fir-md1-s1 kernel: LNet: Service thread pid 97664 completed after 472.04s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jul 03 17:04:17 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 17:04:17 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 751 previous similar messages Jul 03 17:09:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 86b912bf-2e5b-c1ac-9553-f5e705cfca02 (at 10.9.106.51@o2ib4) Jul 03 17:09:55 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 03 17:14:18 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 03 17:14:18 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 608 previous similar messages Jul 03 17:22:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 98a67850-1b7c-ef40-1816-b3372d04b91a (at 10.9.104.26@o2ib4) Jul 03 17:22:11 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 17:24:18 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 17:24:18 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 664 previous similar messages Jul 03 17:27:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9d122243-83ef-341e-1d9e-5ad0fa272beb (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f252ffd3400, cur 1562200067 expire 1562199917 last 1562199840 Jul 03 17:27:47 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 17:32:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to df993956-2257-9a73-35ef-341b2f75d156 (at 10.9.106.58@o2ib4) Jul 03 17:32:13 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 03 17:34:19 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 17:34:19 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 842 previous similar messages Jul 03 17:44:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 86b912bf-2e5b-c1ac-9553-f5e705cfca02 (at 10.9.106.51@o2ib4) Jul 03 17:44:07 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 03 17:44:21 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 17:44:21 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 573 previous similar messages Jul 03 17:51:59 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jul 03 17:54:28 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 17:54:28 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 743 previous similar messages Jul 03 17:54:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to f174f128-4488-2485-c92d-799c5cc7f49d (at 10.9.104.27@o2ib4) Jul 03 17:54:58 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 03 18:04:47 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 18:04:47 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 869 previous similar messages Jul 03 18:05:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 7cdd6fe1-f6f2-0a49-df73-de49ebbd85ff (at 10.9.101.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fd146400, cur 1562202309 expire 1562202159 last 1562202082 Jul 03 18:05:09 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 18:14:47 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 18:14:47 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 607 previous similar messages Jul 03 18:24:55 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 18:24:55 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 763 previous similar messages Jul 03 18:31:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to bff84d1e-0a69-b6c4-379f-b22c9974d598 (at 10.9.114.3@o2ib4) Jul 03 18:31:25 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 03 18:33:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02ea0e3d-c72b-2664-4a33-3841a13fb806 (at 10.9.101.55@o2ib4) Jul 03 18:33:18 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 18:35:44 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 18:35:44 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 864 previous similar messages Jul 03 18:46:03 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 18:46:03 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 638 previous similar messages Jul 03 18:56:12 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 18:56:12 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 811 previous similar messages Jul 03 19:06:21 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 19:06:21 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 609 previous similar messages Jul 03 19:16:46 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 19:16:46 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 804 previous similar messages Jul 03 19:27:47 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 19:27:47 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 887 previous similar messages Jul 03 19:37:58 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 19:37:58 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 651 previous similar messages Jul 03 19:47:58 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 19:47:58 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 803 previous similar messages Jul 03 19:52:25 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 03 19:52:30 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 03 19:52:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 36c50ebf-42f1-2e51-f789-02d6d7eec692 (at 10.8.8.33@o2ib6) reconnecting Jul 03 19:52:31 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 03 19:52:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 347ffbdc-328a-c7b5-0dc8-6a73375f2e66 (at 10.8.8.33@o2ib6) Jul 03 19:52:31 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 19:52:36 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 03 19:52:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to fba120c3-4cd2-22a3-cb05-96d005aa975a (at 10.8.21.2@o2ib6) Jul 03 19:52:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.21.2@o2ib6, removing former export from same NID Jul 03 19:52:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.27.19@o2ib6, removing former export from same NID Jul 03 19:52:58 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 03 19:53:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.7.19@o2ib6, removing former export from same NID Jul 03 19:53:00 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 03 19:53:01 fir-md1-s1 kernel: Lustre: 24578:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1562208773/real 0] req@ffff8f1b6671cb00 x1636723431349072/t0(0) o106->fir-MDT0000@10.8.28.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562208781 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 19:53:01 fir-md1-s1 kernel: Lustre: 24578:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 03 19:53:04 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 03 19:53:04 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Jul 03 19:53:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.25.12@o2ib6, removing former export from same NID Jul 03 19:53:05 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 03 19:53:05 fir-md1-s1 kernel: LustreError: 44037:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8f21c9b4f450 x1635707640078096/t0(0) o3->4ed462a8-ed6a-0891-ced6-ebadfda1f88d@10.8.8.30@o2ib6:27/0 lens 488/440 e 0 to 0 dl 1562208807 ref 1 fl Interpret:/0/0 rc 0/0 Jul 03 19:53:06 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1eb5573a00 Jul 03 19:53:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 4ed462a8-ed6a-0891-ced6-ebadfda1f88d (at 10.8.8.30@o2ib6), client will retry: rc -110 Jul 03 19:53:08 fir-md1-s1 kernel: Lustre: 26256:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f22f020dd00 x1638067932108544/t0(0) o101->b041cef5-fff9-4fc6-cc5f-62c5a80e124b@10.9.0.81@o2ib4:13/0 lens 480/568 e 1 to 0 dl 1562208793 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 19:53:08 fir-md1-s1 kernel: Lustre: 26256:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 03 19:53:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.10@o2ib6, removing former export from same NID Jul 03 19:53:15 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 03 19:53:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 469d3c01-0ba5-8df1-fade-b379f197d2fe (at 10.8.27.33@o2ib6), client will retry: rc = -110 Jul 03 19:53:26 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 19:53:27 fir-md1-s1 kernel: Lustre: 22004:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1562208796/real 0] req@ffff8f1e2f262700 x1636723431389552/t0(0) o104->fir-MDT0002@10.8.8.33@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562208807 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 19:53:27 fir-md1-s1 kernel: Lustre: 22004:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 03 19:53:29 fir-md1-s1 kernel: Lustre: 97644:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1d1c1daa00 x1633726786217616/t0(0) o101->23504e9e-38b0-73ab-6845-a2f9362c9ca3@10.8.29.7@o2ib6:4/0 lens 480/568 e 0 to 0 dl 1562208814 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 19:53:33 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 03 19:53:33 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Jul 03 19:53:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.16@o2ib6, removing former export from same NID Jul 03 19:53:34 fir-md1-s1 kernel: Lustre: Skipped 501 previous similar messages Jul 03 19:53:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to de0940aa-281f-ee72-6d66-43860c09ff15 (at 10.8.17.16@o2ib6) Jul 03 19:53:34 fir-md1-s1 kernel: Lustre: Skipped 718 previous similar messages Jul 03 19:53:43 fir-md1-s1 kernel: LustreError: 46531:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f2005584850 x1633726786220736/t0(0) o4->23504e9e-38b0-73ab-6845-a2f9362c9ca3@10.8.29.7@o2ib6:3/0 lens 488/448 e 0 to 0 dl 1562208843 ref 1 fl Interpret:/0/0 rc 0/0 Jul 03 19:53:43 fir-md1-s1 kernel: LustreError: 46531:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message Jul 03 19:53:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client c977be3c-f98f-fbec-3aac-245ba5109971 (at 10.8.30.35@o2ib6) reconnecting Jul 03 19:53:46 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 03 19:53:47 fir-md1-s1 kernel: LustreError: 46526:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f2005585450 x1634456217841056/t0(0) o4->b95afc0f-d5ce-0d5e-e5e9-03cd8d169d60@10.8.8.12@o2ib6:17/0 lens 504/448 e 1 to 0 dl 1562208827 ref 1 fl Interpret:/2/0 rc 0/0 Jul 03 19:53:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with b95afc0f-d5ce-0d5e-e5e9-03cd8d169d60 (at 10.8.8.12@o2ib6), client will retry: rc = -110 Jul 03 19:53:50 fir-md1-s1 kernel: Lustre: 22004:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:4s); client may timeout. req@ffff8f203b635400 x1634928116944112/t349864063350(0) o101->36c50ebf-42f1-2e51-f789-02d6d7eec692@10.8.8.33@o2ib6:16/0 lens 376/944 e 0 to 0 dl 1562208826 ref 1 fl Complete:/0/0 rc 0/0 Jul 03 19:53:52 fir-md1-s1 kernel: Lustre: 20723:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:5s); client may timeout. req@ffff8f20e8993300 x1633726786220496/t0(0) o101->23504e9e-38b0-73ab-6845-a2f9362c9ca3@10.8.29.7@o2ib6:17/0 lens 480/536 e 0 to 0 dl 1562208827 ref 1 fl Complete:/0/0 rc 0/0 Jul 03 19:53:54 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f22d5a19e00 Jul 03 19:53:54 fir-md1-s1 kernel: Lustre: 46591:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:3s); client may timeout. req@ffff8f217734f850 x1634525650098400/t0(0) o4->2ee51d45-426d-bbd9-5b4f-485a0917e8b9@10.8.17.18@o2ib6:21/0 lens 504/448 e 1 to 0 dl 1562208831 ref 1 fl Complete:/0/ffffffff rc -110/-1 Jul 03 19:53:58 fir-md1-s1 kernel: LustreError: 44037:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f217734bc50 x1633733160145008/t0(0) o4->00a6bf4a-1a11-675b-07eb-2392e93c70c7@10.8.29.8@o2ib6:28/0 lens 488/448 e 1 to 0 dl 1562208838 ref 1 fl Interpret:/2/0 rc 0/0 Jul 03 19:53:58 fir-md1-s1 kernel: LustreError: 44037:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message Jul 03 19:54:00 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f181c0a8600 Jul 03 19:54:02 fir-md1-s1 kernel: Lustre: 46562:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1c43eee050 x1631306790892784/t0(0) o3->6e0b1c17-2142-9190-acc8-624208298012@10.8.8.17@o2ib6:7/0 lens 488/440 e 0 to 0 dl 1562208847 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 19:54:02 fir-md1-s1 kernel: Lustre: 46562:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 15 previous similar messages Jul 03 19:54:04 fir-md1-s1 kernel: Lustre: 97665:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:11s); client may timeout. req@ffff8f17c3d7dd00 x1633726786220688/t0(0) o101->23504e9e-38b0-73ab-6845-a2f9362c9ca3@10.8.29.7@o2ib6:23/0 lens 480/536 e 0 to 0 dl 1562208833 ref 1 fl Complete:/0/0 rc 0/0 Jul 03 19:54:05 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 03 19:54:05 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages Jul 03 19:54:05 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f159e9c0000 Jul 03 19:54:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO write error with 00a6bf4a-1a11-675b-07eb-2392e93c70c7 (at 10.8.29.8@o2ib6), client will retry: rc = -110 Jul 03 19:54:05 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 19:54:05 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d89fd7400 Jul 03 19:54:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.15.6@o2ib6, removing former export from same NID Jul 03 19:54:11 fir-md1-s1 kernel: Lustre: Skipped 350 previous similar messages Jul 03 19:54:14 fir-md1-s1 kernel: Lustre: 97648:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1562208847/real 0] req@ffff8f1c368e0600 x1636723431470800/t0(0) o104->fir-MDT0002@10.8.1.33@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562208854 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 19:54:14 fir-md1-s1 kernel: LustreError: 21865:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 20+0s req@ffff8f1c18195d00 x1637979061666736/t0(0) o37->4dda764c-5ca7-3340-a1d3-17b756c64805@10.8.0.67@o2ib6:14/0 lens 448/440 e 1 to 0 dl 1562208854 ref 1 fl Interpret:/0/0 rc 0/0 Jul 03 19:54:14 fir-md1-s1 kernel: Lustre: 97648:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 03 19:54:15 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1f04d00400 Jul 03 19:54:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 9eed212b-34d9-6e26-f1ac-cdc452decf97 (at 10.8.29.3@o2ib6), client will retry: rc -110 Jul 03 19:54:21 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f16630c0a00 Jul 03 19:54:21 fir-md1-s1 kernel: Lustre: 21865:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:7s); client may timeout. req@ffff8f1c18195d00 x1637979061666736/t0(0) o37->4dda764c-5ca7-3340-a1d3-17b756c64805@10.8.0.67@o2ib6:14/0 lens 448/408 e 1 to 0 dl 1562208854 ref 1 fl Complete:/0/0 rc -110/-110 Jul 03 19:54:21 fir-md1-s1 kernel: Lustre: 21865:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages Jul 03 19:54:23 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1cca9d3a00 Jul 03 19:54:27 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20c124b400 Jul 03 19:54:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 12e474d9-b4d9-2c7f-2e45-e7d8f457f930 (at 10.8.16.8@o2ib6), client will retry: rc -110 Jul 03 19:54:27 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f195da25200 Jul 03 19:54:28 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e59365600 Jul 03 19:54:29 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f17cc6f9c00 Jul 03 19:54:29 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1bff13f800 Jul 03 19:54:33 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1dcbf9c000 Jul 03 19:54:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 6e0b1c17-2142-9190-acc8-624208298012 (at 10.8.8.17@o2ib6), client will retry: rc -110 Jul 03 19:54:33 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 03 19:54:35 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d1d2ede00 Jul 03 19:54:37 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1f9ba66600 Jul 03 19:54:41 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e5926d800 Jul 03 19:54:44 fir-md1-s1 kernel: Lustre: 20723:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:21s); client may timeout. req@ffff8f1b6671a100 x1631547122810496/t349864079075(0) o101->a5eec2e6-62e8-19e2-7ed8-f567dc50fbb0@10.8.8.32@o2ib6:23/0 lens 416/944 e 0 to 0 dl 1562208863 ref 1 fl Complete:/0/0 rc 0/0 Jul 03 19:54:44 fir-md1-s1 kernel: Lustre: 20723:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jul 03 19:54:45 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 56s: evicting client at 10.8.2.20@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f224e8b69c0/0x5d9ee62ec5bd374f lrc: 3/0,0 mode: PW/PW res: [0x2c002be96:0x4918:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.8.2.20@o2ib6 remote: 0xbfd4fbab82a26d7b expref: 2292 pid: 97648 timeout: 1323945 lvb_type: 0 Jul 03 19:54:47 fir-md1-s1 kernel: LustreError: 21388:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f1f5875bc50 x1631552256687440/t0(0) o4->8167f9b2-58bb-1a00-523a-9433a074fe32@10.8.27.28@o2ib6:17/0 lens 520/456 e 1 to 0 dl 1562208887 ref 1 fl Interpret:/0/0 rc 0/0 Jul 03 19:54:47 fir-md1-s1 kernel: LustreError: 21388:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message Jul 03 19:54:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 6c3e9cd1-a2aa-e356-67b4-60b86ef1d3c6 (at 10.8.16.6@o2ib6) Jul 03 19:54:49 fir-md1-s1 kernel: Lustre: Skipped 956 previous similar messages Jul 03 19:54:49 fir-md1-s1 kernel: LustreError: 20367:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.2.20@o2ib6 arrived at 1562208889 with bad export cookie 6746082289093297427 Jul 03 19:54:49 fir-md1-s1 kernel: LustreError: 20367:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 2 previous similar messages Jul 03 19:54:49 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1a05616000 Jul 03 19:54:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 8167f9b2-58bb-1a00-523a-9433a074fe32 (at 10.8.27.28@o2ib6), client will retry: rc = -110 Jul 03 19:54:49 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 19:54:51 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f196655e800 Jul 03 19:55:04 fir-md1-s1 kernel: LustreError: 22009:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.2.20@o2ib6 arrived at 1562208904 with bad export cookie 6746082289093297427 Jul 03 19:55:11 fir-md1-s1 kernel: Lustre: 46591:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1852ca5050 x1634456217841056/t0(0) o4->b95afc0f-d5ce-0d5e-e5e9-03cd8d169d60@10.8.8.12@o2ib6:16/0 lens 504/448 e 1 to 0 dl 1562208916 ref 2 fl Interpret:/2/0 rc 0/0 Jul 03 19:55:11 fir-md1-s1 kernel: Lustre: 46591:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 35 previous similar messages Jul 03 19:55:12 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 03 19:55:12 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 17 previous similar messages Jul 03 19:55:16 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1ea5afc400 Jul 03 19:55:18 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1bf4434e00 Jul 03 19:55:20 fir-md1-s1 kernel: LustreError: 46560:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f1852ca3050 x1634525650098400/t0(0) o4->2ee51d45-426d-bbd9-5b4f-485a0917e8b9@10.8.17.18@o2ib6:20/0 lens 504/448 e 1 to 0 dl 1562208920 ref 1 fl Interpret:/2/0 rc 0/0 Jul 03 19:55:20 fir-md1-s1 kernel: LustreError: 46560:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 3 previous similar messages Jul 03 19:55:20 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f243d944600 Jul 03 19:55:21 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f24407e5400 Jul 03 19:55:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.18.20@o2ib6, removing former export from same NID Jul 03 19:55:26 fir-md1-s1 kernel: Lustre: Skipped 1066 previous similar messages Jul 03 19:55:30 fir-md1-s1 kernel: Lustre: 97648:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:5s); client may timeout. req@ffff8f172aad6600 x1631555520463888/t0(0) o101->d36980b7-2b04-f724-0e6b-cf989e4d7da2@10.8.1.34@o2ib6:25/0 lens 480/536 e 0 to 0 dl 1562208925 ref 1 fl Complete:/0/0 rc 0/0 Jul 03 19:55:30 fir-md1-s1 kernel: Lustre: 97648:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages Jul 03 19:55:33 fir-md1-s1 kernel: Lustre: 23748:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1562208926/real 0] req@ffff8f2d3aa67200 x1636723431595232/t0(0) o104->fir-MDT0000@10.8.0.66@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562208933 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 19:55:33 fir-md1-s1 kernel: Lustre: 23748:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 03 19:55:51 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 5f3b8986-88bc-dd5d-4c41-5670b4e69c0b (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0e3253a000, cur 1562208951 expire 1562208801 last 1562208724 Jul 03 19:55:51 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 03 19:55:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 03 19:56:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 03 19:56:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 2c0bfc93-71cb-f565-f1fb-8f804a23ec4c (at 10.8.1.26@o2ib6) reconnecting Jul 03 19:56:20 fir-md1-s1 kernel: Lustre: Skipped 235 previous similar messages Jul 03 19:57:25 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 03 19:57:25 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 24 previous similar messages Jul 03 19:57:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 837c124c-41d9-368d-aae3-f10235137c33 (at 10.8.18.3@o2ib6) Jul 03 19:57:44 fir-md1-s1 kernel: Lustre: Skipped 1032 previous similar messages Jul 03 19:57:47 fir-md1-s1 kernel: Lustre: 22282:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1562209055/real 0] req@ffff8f41b680f500 x1636723431843824/t0(0) o104->fir-MDT0002@10.8.2.20@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562209066 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 19:57:47 fir-md1-s1 kernel: Lustre: 22282:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 03 19:57:50 fir-md1-s1 kernel: LustreError: 21708:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f1852ca2c50 x1635707640142560/t0(0) o4->4ed462a8-ed6a-0891-ced6-ebadfda1f88d@10.8.8.30@o2ib6:11/0 lens 488/448 e 0 to 0 dl 1562209091 ref 1 fl Interpret:/0/0 rc 0/0 Jul 03 19:57:50 fir-md1-s1 kernel: LustreError: 21708:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 11 previous similar messages Jul 03 19:57:59 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f22c3abd000 Jul 03 19:57:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Bulk IO read error with 9dcf2f2b-339d-b96d-0792-e79b27f28969 (at 10.8.28.2@o2ib6), client will retry: rc -110 Jul 03 19:57:59 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 03 19:57:59 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f16e282ac00 Jul 03 19:58:01 fir-md1-s1 kernel: Lustre: 97643:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f16666f3000 x1635707640142496/t0(0) o101->4ed462a8-ed6a-0891-ced6-ebadfda1f88d@10.8.8.30@o2ib6:6/0 lens 376/976 e 0 to 0 dl 1562209086 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 19:58:01 fir-md1-s1 kernel: Lustre: 97643:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 17 previous similar messages Jul 03 19:58:03 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1b7c255a00 Jul 03 19:58:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with 4ed462a8-ed6a-0891-ced6-ebadfda1f88d (at 10.8.8.30@o2ib6), client will retry: rc = -110 Jul 03 19:58:03 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 03 19:58:10 fir-md1-s1 kernel: LustreError: 46535:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 20+0s req@ffff8f1852ca3c50 x1635200122088912/t0(0) o3->018b4088-9100-7f5b-2709-38dd7f461ac7@10.8.8.29@o2ib6:10/0 lens 488/440 e 1 to 0 dl 1562209090 ref 1 fl Interpret:/0/0 rc 0/0 Jul 03 19:58:10 fir-md1-s1 kernel: LustreError: 46535:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message Jul 03 19:58:19 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e470e5a00 Jul 03 19:58:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with aba5d4eb-e07c-9b0f-6ab5-7f97caf38a26 (at 10.8.16.4@o2ib6), client will retry: rc -110 Jul 03 19:58:20 fir-md1-s1 kernel: Lustre: 46532:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:2s); client may timeout. req@ffff8f1852ca7c50 x1631583824738768/t0(0) o3->aba5d4eb-e07c-9b0f-6ab5-7f97caf38a26@10.8.16.4@o2ib6:17/0 lens 488/440 e 0 to 0 dl 1562209097 ref 1 fl Complete:/0/ffffffff rc -110/-1 Jul 03 19:58:20 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 44s: evicting client at 10.8.8.30@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f346d2a3600/0x5d9ee62ec6a33ef9 lrc: 4/0,0 mode: EX/EX res: [0x2c002bedb:0xeec5:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT flags: 0x60000400000020 nid: 10.8.8.30@o2ib6 remote: 0x44bc588de19b9b76 expref: 14320 pid: 97643 timeout: 1324160 lvb_type: 3 Jul 03 19:58:21 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1a142ea600 Jul 03 19:58:23 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 19:58:23 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 643 previous similar messages Jul 03 19:58:23 fir-md1-s1 kernel: LustreError: 83752:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f06f27ef800 x1636723432054432/t0(0) o105->fir-MDT0002@10.8.8.30@o2ib6:15/16 lens 304/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 03 19:58:23 fir-md1-s1 kernel: LustreError: 83752:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jul 03 19:58:24 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1885204000 Jul 03 19:58:25 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d007c1200 Jul 03 19:58:25 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f21c382dc00 Jul 03 19:58:25 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f16627d1200 Jul 03 19:58:27 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1a31af6800 Jul 03 19:58:27 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1a9fcffc00 Jul 03 19:58:29 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1d1c1da200 Jul 03 19:58:29 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20cb0c3600 Jul 03 19:58:29 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f17fa734c00 Jul 03 19:58:30 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20de015a00 Jul 03 19:58:30 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20d73fd800 Jul 03 19:58:30 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1e341d8000 Jul 03 19:58:30 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f251f04fc00 Jul 03 19:58:32 fir-md1-s1 kernel: LustreError: 20188:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f20f3471800 Jul 03 19:58:33 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f173e89b200 Jul 03 19:58:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.9.9@o2ib6, removing former export from same NID Jul 03 19:58:41 fir-md1-s1 kernel: Lustre: Skipped 309 previous similar messages Jul 03 19:58:41 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2048977200 Jul 03 19:58:49 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f23a926b200 Jul 03 19:58:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 03 19:58:53 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2318d27200 Jul 03 19:58:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with 12e474d9-b4d9-2c7f-2e45-e7d8f457f930 (at 10.8.16.8@o2ib6), client will retry: rc -110 Jul 03 19:58:53 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 03 19:59:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 03 20:00:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 03 20:08:44 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 20:08:44 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 797 previous similar messages Jul 03 20:19:03 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 20:19:03 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 832 previous similar messages Jul 03 20:29:10 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 20:29:10 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 623 previous similar messages Jul 03 20:40:56 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 03 20:40:56 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 735 previous similar messages Jul 03 20:51:02 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 20:51:02 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 805 previous similar messages Jul 03 21:01:05 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 21:01:05 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 843 previous similar messages Jul 03 21:11:12 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 21:11:12 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 623 previous similar messages Jul 03 21:21:15 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 21:21:15 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 807 previous similar messages Jul 03 21:32:27 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 21:32:27 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 724 previous similar messages Jul 03 21:42:48 fir-md1-s1 kernel: LustreError: 21289:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 21:42:48 fir-md1-s1 kernel: LustreError: 21289:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 795 previous similar messages Jul 03 21:52:52 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 21:52:52 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 853 previous similar messages Jul 03 22:03:15 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 22:03:15 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 610 previous similar messages Jul 03 22:13:19 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 22:13:19 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 859 previous similar messages Jul 03 22:22:44 fir-md1-s1 kernel: Lustre: 50446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562217757/real 1562217757] req@ffff8f1fca084200 x1636723449609552/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562217764 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 03 22:22:44 fir-md1-s1 kernel: Lustre: 50446:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 03 22:22:52 fir-md1-s1 kernel: Lustre: 26255:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f19e8a6cb00 x1631600744376720/t0(0) o36->40db60e6-2b5f-e52d-2610-43b84e2f829d@10.8.29.1@o2ib6:27/0 lens 496/448 e 1 to 0 dl 1562217777 ref 2 fl Interpret:/0/0 rc 0/0 Jul 03 22:22:52 fir-md1-s1 kernel: Lustre: 26255:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 22 previous similar messages Jul 03 22:22:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 40db60e6-2b5f-e52d-2610-43b84e2f829d (at 10.8.29.1@o2ib6) reconnecting Jul 03 22:22:58 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 03 22:22:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 6e32fe6b-eec6-274e-37cd-da661cf9bf17 (at 10.8.29.1@o2ib6) Jul 03 22:22:58 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 03 22:23:19 fir-md1-s1 kernel: Lustre: 50446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562217792/real 1562217792] req@ffff8f1fca084200 x1636723449609552/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562217799 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 22:23:19 fir-md1-s1 kernel: Lustre: 50446:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 03 22:23:33 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 22:23:33 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 619 previous similar messages Jul 03 22:23:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 40db60e6-2b5f-e52d-2610-43b84e2f829d (at 10.8.29.1@o2ib6) reconnecting Jul 03 22:23:40 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 03 22:23:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 6e32fe6b-eec6-274e-37cd-da661cf9bf17 (at 10.8.29.1@o2ib6) Jul 03 22:23:40 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 03 22:24:07 fir-md1-s1 kernel: LustreError: 97650:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562217757, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1d73512d00/0x5d9ee62f0ba095df lrc: 3/1,0 mode: --/PR res: [0x2000297d4:0x4a2:0x0].0x0 bits 0x13/0x0 rrc: 25 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97650 timeout: 0 lvb_type: 0 Jul 03 22:24:07 fir-md1-s1 kernel: LustreError: 97650:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 03 22:24:08 fir-md1-s1 kernel: LustreError: 20460:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562217758, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1c3d2a8000/0x5d9ee62f0ba69689 lrc: 3/1,0 mode: --/PR res: [0x2000297d4:0x4a2:0x0].0x0 bits 0x13/0x0 rrc: 25 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20460 timeout: 0 lvb_type: 0 Jul 03 22:24:11 fir-md1-s1 kernel: LustreError: 97643:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562217761, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2531dfad00/0x5d9ee62f0bb92624 lrc: 3/1,0 mode: --/PR res: [0x2000297d4:0x4a2:0x0].0x0 bits 0x13/0x0 rrc: 25 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97643 timeout: 0 lvb_type: 0 Jul 03 22:24:17 fir-md1-s1 kernel: LustreError: 23567:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562217767, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f051862b180/0x5d9ee62f0bd2a6b5 lrc: 3/1,0 mode: --/PR res: [0x2000297d4:0x4a2:0x0].0x0 bits 0x13/0x0 rrc: 25 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23567 timeout: 0 lvb_type: 0 Jul 03 22:24:17 fir-md1-s1 kernel: LustreError: 23567:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 03 22:24:25 fir-md1-s1 kernel: LustreError: 20728:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562217775, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f159eea0fc0/0x5d9ee62f0bfab832 lrc: 3/1,0 mode: --/PR res: [0x2000297d4:0x4a2:0x0].0x0 bits 0x13/0x0 rrc: 27 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20728 timeout: 0 lvb_type: 0 Jul 03 22:24:25 fir-md1-s1 kernel: LustreError: 20728:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages Jul 03 22:24:29 fir-md1-s1 kernel: Lustre: 50446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562217862/real 1562217862] req@ffff8f1fca084200 x1636723449609552/t0(0) o104->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562217869 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 03 22:24:29 fir-md1-s1 kernel: Lustre: 50446:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 03 22:24:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 0f8f808f-b03b-81e6-e30e-46ff547f2e45 (at 10.9.113.3@o2ib4) reconnecting Jul 03 22:24:59 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 03 22:24:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 8651a829-1584-35b1-6264-26a8d5433bb6 (at 10.9.113.3@o2ib4) Jul 03 22:24:59 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 03 22:25:11 fir-md1-s1 kernel: LustreError: 50446:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.15.6@o2ib6) failed to reply to blocking AST (req@ffff8f1fca084200 x1636723449609552 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f17bdf31440/0x5d9ee62f0acb4901 lrc: 4/0,0 mode: PR/PR res: [0x2000297d4:0x4a2:0x0].0x0 bits 0x13/0x0 rrc: 31 type: IBT flags: 0x60200400000020 nid: 10.8.15.6@o2ib6 remote: 0x71e36d96c02791d expref: 16638 pid: 21460 timeout: 1333113 lvb_type: 0 Jul 03 22:25:11 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.15.6@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 03 22:25:11 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.15.6@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f17bdf31440/0x5d9ee62f0acb4901 lrc: 3/0,0 mode: PR/PR res: [0x2000297d4:0x4a2:0x0].0x0 bits 0x13/0x0 rrc: 31 type: IBT flags: 0x60200400000020 nid: 10.8.15.6@o2ib6 remote: 0x71e36d96c02791d expref: 16639 pid: 21460 timeout: 0 lvb_type: 0 Jul 03 22:26:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bde0b95a-d079-f6a9-2817-38a3e98f4627 (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1615f91c00, cur 1562217966 expire 1562217816 last 1562217739 Jul 03 22:33:56 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 22:33:56 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 800 previous similar messages Jul 03 22:44:04 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 22:44:04 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 634 previous similar messages Jul 03 22:54:16 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 22:54:16 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 782 previous similar messages Jul 03 23:04:30 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 23:04:30 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 673 previous similar messages Jul 03 23:14:33 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 23:14:33 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 802 previous similar messages Jul 03 23:24:39 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 23:24:39 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 623 previous similar messages Jul 03 23:34:39 fir-md1-s1 kernel: LustreError: 46591:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 23:34:39 fir-md1-s1 kernel: LustreError: 46591:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 793 previous similar messages Jul 03 23:39:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5f4bee65-bf6b-ad1e-3c5b-2158af12057b (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25219b3800, cur 1562222387 expire 1562222237 last 1562222160 Jul 03 23:39:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 03 23:44:44 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 23:44:44 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 556 previous similar messages Jul 03 23:54:44 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 03 23:54:44 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 856 previous similar messages Jul 04 00:04:59 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 00:04:59 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 562 previous similar messages Jul 04 00:15:03 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 00:15:03 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 860 previous similar messages Jul 04 00:25:16 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 00:25:16 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 565 previous similar messages Jul 04 00:36:17 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 00:36:17 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 873 previous similar messages Jul 04 00:46:39 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 00:46:39 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 644 previous similar messages Jul 04 00:56:45 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 00:56:45 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 817 previous similar messages Jul 04 01:07:05 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 01:07:05 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 604 previous similar messages Jul 04 01:14:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 07c1712c-9739-2dce-4883-ed8d604a7bd1 (at 10.8.15.3@o2ib6) reconnecting Jul 04 01:14:02 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 04 01:14:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 420c129b-df9e-b1c5-eae5-667fed64bb9d (at 10.8.15.3@o2ib6) Jul 04 01:14:02 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 04 01:14:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 07c1712c-9739-2dce-4883-ed8d604a7bd1 (at 10.8.15.3@o2ib6) reconnecting Jul 04 01:14:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 420c129b-df9e-b1c5-eae5-667fed64bb9d (at 10.8.15.3@o2ib6) Jul 04 01:14:45 fir-md1-s1 kernel: Lustre: 22004:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f189e7a2a00 x1631537913284992/t0(0) o101->d3013375-2e90-b76e-c4d8-76867f2b4a32@10.8.2.20@o2ib6:20/0 lens 480/568 e 1 to 0 dl 1562228090 ref 2 fl Interpret:/0/0 rc 0/0 Jul 04 01:14:45 fir-md1-s1 kernel: Lustre: 22004:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages Jul 04 01:17:09 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 01:17:09 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 794 previous similar messages Jul 04 01:27:24 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 01:27:24 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 565 previous similar messages Jul 04 01:37:38 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 01:37:38 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 808 previous similar messages Jul 04 01:48:41 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 01:48:41 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 923 previous similar messages Jul 04 01:58:53 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 01:58:53 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 645 previous similar messages Jul 04 02:09:10 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 02:09:10 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 853 previous similar messages Jul 04 02:19:16 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 02:19:16 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 617 previous similar messages Jul 04 02:29:34 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 02:29:34 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 806 previous similar messages Jul 04 02:39:58 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 02:39:58 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 615 previous similar messages Jul 04 02:50:06 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 02:50:06 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 859 previous similar messages Jul 04 03:00:24 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 03:00:24 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 606 previous similar messages Jul 04 03:11:19 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 03:11:19 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 884 previous similar messages Jul 04 03:21:26 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 03:21:26 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 653 previous similar messages Jul 04 03:31:27 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 03:31:27 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 583 previous similar messages Jul 04 03:41:51 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 03:41:51 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 868 previous similar messages Jul 04 03:51:53 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 03:51:53 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 629 previous similar messages Jul 04 04:02:09 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 04:02:09 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 835 previous similar messages Jul 04 04:12:27 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 04:12:27 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 617 previous similar messages Jul 04 04:22:30 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 04:22:30 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 819 previous similar messages Jul 04 04:32:31 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 04:32:31 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 588 previous similar messages Jul 04 04:44:07 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 04:44:07 fir-md1-s1 kernel: LustreError: 21451:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 738 previous similar messages Jul 04 04:54:15 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 04:54:15 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 797 previous similar messages Jul 04 05:04:17 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 05:04:17 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 649 previous similar messages Jul 04 05:14:19 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 05:14:19 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 792 previous similar messages Jul 04 05:24:45 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 05:24:45 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 621 previous similar messages Jul 04 05:35:23 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 05:35:23 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 833 previous similar messages Jul 04 05:45:30 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 05:45:30 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 661 previous similar messages Jul 04 05:57:14 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 32768 GRANT, real grant 0 Jul 04 05:57:14 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 780 previous similar messages Jul 04 06:07:21 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 06:07:21 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 793 previous similar messages Jul 04 06:17:42 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 06:17:42 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 594 previous similar messages Jul 04 06:28:08 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 06:28:08 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 849 previous similar messages Jul 04 06:38:32 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 06:38:32 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 600 previous similar messages Jul 04 06:48:56 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 06:48:56 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 622 previous similar messages Jul 04 06:59:00 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 06:59:00 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 846 previous similar messages Jul 04 07:09:08 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 07:09:08 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 569 previous similar messages Jul 04 07:19:17 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 07:19:17 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 764 previous similar messages Jul 04 07:29:30 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 07:29:30 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 691 previous similar messages Jul 04 07:39:31 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 07:39:31 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 572 previous similar messages Jul 04 07:49:37 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 07:49:37 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 552 previous similar messages Jul 04 07:59:51 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 07:59:51 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 617 previous similar messages Jul 04 08:09:54 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 08:09:54 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 625 previous similar messages Jul 04 08:20:33 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 32768 GRANT, real grant 0 Jul 04 08:20:33 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 713 previous similar messages Jul 04 08:32:34 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 08:32:34 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 730 previous similar messages Jul 04 08:44:17 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 08:44:17 fir-md1-s1 kernel: LustreError: 46555:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 727 previous similar messages Jul 04 08:56:06 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 08:56:06 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 727 previous similar messages Jul 04 08:59:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 25bbf676-f42f-a624-a39b-ff8deef07eff (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2528bcb800, cur 1562255942 expire 1562255792 last 1562255715 Jul 04 08:59:02 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 04 08:59:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 25bbf676-f42f-a624-a39b-ff8deef07eff (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1ceb2b4800, cur 1562255947 expire 1562255797 last 1562255720 Jul 04 08:59:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 04 08:59:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ec76f1db-9c9b-bbe0-847f-90a9d517c8dc (at 10.8.9.8@o2ib6) Jul 04 09:06:23 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 09:06:23 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 769 previous similar messages Jul 04 09:17:13 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 09:17:13 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 659 previous similar messages Jul 04 09:27:17 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 09:27:17 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 643 previous similar messages Jul 04 09:37:41 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 09:37:41 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 567 previous similar messages Jul 04 09:49:02 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 09:49:02 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 718 previous similar messages Jul 04 09:59:04 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 09:59:04 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 797 previous similar messages Jul 04 10:09:23 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 10:09:23 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 591 previous similar messages Jul 04 10:19:44 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 10:19:44 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 711 previous similar messages Jul 04 10:37:51 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 10:37:51 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 720 previous similar messages Jul 04 10:47:52 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 10:47:52 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 456 previous similar messages Jul 04 10:57:55 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 10:57:55 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 384 previous similar messages Jul 04 11:10:11 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 11:10:11 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 197 previous similar messages Jul 04 11:20:12 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 11:20:12 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 403 previous similar messages Jul 04 11:30:16 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 11:30:16 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 367 previous similar messages Jul 04 11:40:17 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 11:40:17 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 386 previous similar messages Jul 04 11:50:39 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 11:50:39 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 394 previous similar messages Jul 04 12:00:51 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 12:00:51 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 385 previous similar messages Jul 04 12:08:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 8ef25a02-5cd5-8500-774d-d75ea76eaffd (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3f5ea65400, cur 1562267302 expire 1562267152 last 1562267075 Jul 04 12:10:59 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 12:10:59 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 356 previous similar messages Jul 04 12:21:00 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 12:21:00 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 696 previous similar messages Jul 04 12:31:08 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 12:31:08 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 418 previous similar messages Jul 04 12:41:10 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 04 12:41:10 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 336 previous similar messages Jul 04 12:51:23 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 04 12:51:23 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 279 previous similar messages Jul 04 13:02:58 fir-md1-s1 kernel: LustreError: 69435:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 13:02:58 fir-md1-s1 kernel: LustreError: 69435:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 331 previous similar messages Jul 04 13:13:19 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 04 13:13:19 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 267 previous similar messages Jul 04 13:23:19 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 04 13:23:19 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 364 previous similar messages Jul 04 13:33:19 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 13:33:19 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 452 previous similar messages Jul 04 13:43:26 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 13:43:26 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 461 previous similar messages Jul 04 13:53:26 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 13:53:26 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 487 previous similar messages Jul 04 14:03:29 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 14:03:29 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 659 previous similar messages Jul 04 14:13:33 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 14:13:33 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 445 previous similar messages Jul 04 14:23:34 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 14:23:34 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 460 previous similar messages Jul 04 14:33:35 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 14:33:35 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 502 previous similar messages Jul 04 14:43:40 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 04 14:43:40 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 466 previous similar messages Jul 04 14:53:44 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 14:53:44 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 543 previous similar messages Jul 04 15:03:52 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 15:03:52 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 513 previous similar messages Jul 04 15:13:52 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 15:13:52 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 477 previous similar messages Jul 04 15:23:53 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 15:23:53 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 460 previous similar messages Jul 04 15:33:59 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 04 15:33:59 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 402 previous similar messages Jul 04 15:44:05 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 15:44:05 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 506 previous similar messages Jul 04 15:46:02 fir-md1-s1 kernel: Lustre: 10505:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 15:54:05 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 15:54:05 fir-md1-s1 kernel: LustreError: 21537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 422 previous similar messages Jul 04 16:02:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 26320709-561f-90ed-6684-fea46854b319 (at 10.8.1.29@o2ib6) Jul 04 16:02:31 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 04 16:04:10 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 16:04:10 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 404 previous similar messages Jul 04 16:14:19 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 04 16:14:19 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 372 previous similar messages Jul 04 16:24:22 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 04 16:24:22 fir-md1-s1 kernel: LustreError: 21714:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 377 previous similar messages Jul 04 16:34:29 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 32768 GRANT, real grant 0 Jul 04 16:34:29 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 538 previous similar messages Jul 04 16:41:13 fir-md1-s1 kernel: Lustre: 10589:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 16:44:34 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 16:44:34 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 389 previous similar messages Jul 04 16:46:58 fir-md1-s1 kernel: Lustre: 23625:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 16:46:58 fir-md1-s1 kernel: Lustre: 23625:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages Jul 04 16:54:37 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 04 16:54:37 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 450 previous similar messages Jul 04 17:04:38 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 04 17:04:38 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 528 previous similar messages Jul 04 17:12:26 fir-md1-s1 kernel: Lustre: 10588:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 17:12:26 fir-md1-s1 kernel: Lustre: 10588:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages Jul 04 17:14:43 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 17:14:43 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 518 previous similar messages Jul 04 17:24:37 fir-md1-s1 kernel: Lustre: 23561:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 17:25:00 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 17:25:00 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 588 previous similar messages Jul 04 17:26:18 fir-md1-s1 kernel: Lustre: 23600:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 17:26:18 fir-md1-s1 kernel: Lustre: 23600:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jul 04 17:29:41 fir-md1-s1 kernel: Lustre: 23623:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 17:35:01 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 04 17:35:01 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 515 previous similar messages Jul 04 17:38:43 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.9.104.69@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f3c5a713180/0x5d9ee6305c19a6f9 lrc: 3/0,0 mode: PR/PR res: [0x2c002c23d:0x1c859:0x0].0x0 bits 0x58/0x0 rrc: 3 type: IBT flags: 0x60200400010020 nid: 10.9.104.69@o2ib4 remote: 0xc50f1e8ca834a497 expref: 6755 pid: 23731 timeout: 1402183 lvb_type: 0 Jul 04 17:38:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 8c412c26-542a-dae3-c537-fda210938013 (at 10.9.104.69@o2ib4) Jul 04 17:38:48 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 04 17:43:51 fir-md1-s1 kernel: Lustre: 10505:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 17:43:51 fir-md1-s1 kernel: Lustre: 10505:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages Jul 04 17:45:17 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 04 17:45:17 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 485 previous similar messages Jul 04 17:45:47 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 17:45:47 fir-md1-s1 kernel: Lustre: 21311:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages Jul 04 17:47:38 fir-md1-s1 kernel: Lustre: 23623:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 17:55:17 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 17:55:17 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 521 previous similar messages Jul 04 18:05:20 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 04 18:05:20 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 498 previous similar messages Jul 04 18:13:49 fir-md1-s1 kernel: Lustre: 10589:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 18:15:24 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 18:15:24 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 428 previous similar messages Jul 04 18:16:09 fir-md1-s1 kernel: Lustre: 23672:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 18:16:09 fir-md1-s1 kernel: Lustre: 23672:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 19 previous similar messages Jul 04 18:16:28 fir-md1-s1 kernel: Lustre: 23672:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 18:23:26 fir-md1-s1 kernel: Lustre: 10588:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 18:23:26 fir-md1-s1 kernel: Lustre: 10588:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 91 previous similar messages Jul 04 18:23:50 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 18:23:50 fir-md1-s1 kernel: Lustre: 21370:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages Jul 04 18:24:15 fir-md1-s1 kernel: Lustre: 23653:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 18:24:15 fir-md1-s1 kernel: Lustre: 23653:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 32 previous similar messages Jul 04 18:25:40 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 18:25:40 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 566 previous similar messages Jul 04 18:35:42 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 18:35:42 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 530 previous similar messages Jul 04 18:45:42 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 18:45:42 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 561 previous similar messages Jul 04 18:55:50 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 18:55:50 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 500 previous similar messages Jul 04 19:05:55 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 19:05:55 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 471 previous similar messages Jul 04 19:15:58 fir-md1-s1 kernel: LustreError: 46591:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 19:15:58 fir-md1-s1 kernel: LustreError: 46591:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 437 previous similar messages Jul 04 19:26:24 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 19:26:24 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 348 previous similar messages Jul 04 19:36:24 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 04 19:36:24 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 394 previous similar messages Jul 04 19:46:32 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 19:46:32 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 373 previous similar messages Jul 04 19:52:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1f29c7a2-d2d3-0a98-27b0-578e87d088ab (at 10.8.9.2@o2ib6) Jul 04 19:56:36 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 19:56:36 fir-md1-s1 kernel: LustreError: 46572:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 429 previous similar messages Jul 04 20:06:52 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 20:06:52 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 432 previous similar messages Jul 04 20:16:57 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 20:16:57 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 490 previous similar messages Jul 04 20:26:58 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 20:26:58 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 672 previous similar messages Jul 04 20:37:00 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 04 20:37:00 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 635 previous similar messages Jul 04 20:47:03 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 20:47:03 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 708 previous similar messages Jul 04 20:57:04 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 04 20:57:04 fir-md1-s1 kernel: LustreError: 21245:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 719 previous similar messages Jul 04 21:07:05 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 04 21:07:05 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 610 previous similar messages Jul 04 21:17:09 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 69632 GRANT, real grant 0 Jul 04 21:17:09 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 723 previous similar messages Jul 04 21:24:14 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 721f53d4-652b-e945-12ff-35ccdf15e929 (at 10.9.114.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14c7772800, cur 1562300654 expire 1562300504 last 1562300427 Jul 04 21:24:14 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 04 21:27:11 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 21:27:11 fir-md1-s1 kernel: LustreError: 23106:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 708 previous similar messages Jul 04 21:37:12 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 21:37:12 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 763 previous similar messages Jul 04 21:47:13 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 21:47:13 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 740 previous similar messages Jul 04 21:47:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 721f53d4-652b-e945-12ff-35ccdf15e929 (at 10.9.114.15@o2ib4) Jul 04 21:47:14 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 04 21:57:15 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 21:57:15 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 631 previous similar messages Jul 04 22:07:19 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 22:07:19 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 319 previous similar messages Jul 04 22:17:20 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 22:17:20 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 293 previous similar messages Jul 04 22:27:24 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 22:27:24 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 238 previous similar messages Jul 04 22:37:35 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 22:37:35 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 235 previous similar messages Jul 04 22:44:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6afdb7fb-fdc2-6692-1bb2-94fc70f0b6ac (at 10.9.104.72@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f252253d800, cur 1562305458 expire 1562305308 last 1562305231 Jul 04 22:47:47 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 22:47:47 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 209 previous similar messages Jul 04 22:56:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d4117728-4cc7-9876-91f7-8a96129f589f (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1656e72400, cur 1562306166 expire 1562306016 last 1562305939 Jul 04 22:56:06 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 04 22:57:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c6748fa-faf9-dbf4-7576-e7e488da698d (at 10.8.11.9@o2ib6) Jul 04 22:57:13 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 04 22:57:51 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 22:57:51 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 317 previous similar messages Jul 04 23:08:31 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 23:08:31 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 160 previous similar messages Jul 04 23:14:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c6ff68eb-5fb8-a120-f19a-506df7ae12c5 (at 10.9.104.72@o2ib4) Jul 04 23:14:22 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 04 23:16:45 fir-md1-s1 kernel: Lustre: 10305:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 04 23:16:45 fir-md1-s1 kernel: Lustre: 10305:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 22 previous similar messages Jul 04 23:18:35 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 23:18:35 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 301 previous similar messages Jul 04 23:28:36 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 23:28:36 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 321 previous similar messages Jul 04 23:38:46 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 04 23:38:46 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 304 previous similar messages Jul 04 23:49:09 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 23:49:09 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 216 previous similar messages Jul 04 23:59:11 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 04 23:59:11 fir-md1-s1 kernel: LustreError: 25635:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 260 previous similar messages Jul 05 00:09:15 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 00:09:15 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 248 previous similar messages Jul 05 00:19:16 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 00:19:16 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 249 previous similar messages Jul 05 00:29:17 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 00:29:17 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 223 previous similar messages Jul 05 00:39:18 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 00:39:18 fir-md1-s1 kernel: LustreError: 21539:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 294 previous similar messages Jul 05 00:49:30 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 00:49:30 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 235 previous similar messages Jul 05 00:59:34 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 00:59:34 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 284 previous similar messages Jul 05 01:09:37 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 05 01:09:37 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 297 previous similar messages Jul 05 01:20:05 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 01:20:05 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 316 previous similar messages Jul 05 01:30:06 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 01:30:06 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 306 previous similar messages Jul 05 01:40:11 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 01:40:11 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 275 previous similar messages Jul 05 01:48:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6a859b16-85f7-35c9-387f-f10b0648c129 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25019d5000, cur 1562316520 expire 1562316370 last 1562316293 Jul 05 01:48:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 01:49:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c6748fa-faf9-dbf4-7576-e7e488da698d (at 10.8.11.9@o2ib6) Jul 05 01:49:35 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 01:50:39 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 01:50:39 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 290 previous similar messages Jul 05 02:00:40 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 02:00:40 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 216 previous similar messages Jul 05 02:10:40 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 02:10:40 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 330 previous similar messages Jul 05 02:17:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 3c3d09fd-cece-8f71-77c9-8f6f333d8d68 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f13f73fe000, cur 1562318259 expire 1562318109 last 1562318032 Jul 05 02:17:39 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 02:18:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5c6748fa-faf9-dbf4-7576-e7e488da698d (at 10.8.11.9@o2ib6) Jul 05 02:18:24 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 02:20:42 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 02:20:42 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 330 previous similar messages Jul 05 02:30:44 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 02:30:44 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 246 previous similar messages Jul 05 02:40:46 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 02:40:46 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 269 previous similar messages Jul 05 02:50:51 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 02:50:51 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 224 previous similar messages Jul 05 03:00:56 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 03:00:56 fir-md1-s1 kernel: LustreError: 46520:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 308 previous similar messages Jul 05 03:07:02 fir-md1-s1 kernel: Lustre: 23605:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 05 03:11:04 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 03:11:04 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 205 previous similar messages Jul 05 03:21:05 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 03:21:05 fir-md1-s1 kernel: LustreError: 46543:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 196 previous similar messages Jul 05 03:31:06 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 03:31:06 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 246 previous similar messages Jul 05 03:35:41 fir-md1-s1 kernel: Lustre: 10196:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 05 03:41:09 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 03:41:09 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 179 previous similar messages Jul 05 03:51:16 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 03:51:16 fir-md1-s1 kernel: LustreError: 21686:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 224 previous similar messages Jul 05 04:01:28 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 05 04:01:28 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 242 previous similar messages Jul 05 04:11:36 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 04:11:36 fir-md1-s1 kernel: LustreError: 21567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 233 previous similar messages Jul 05 04:16:57 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 05 04:16:57 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 33 previous similar messages Jul 05 04:17:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 065e2ad2-1d60-8b4d-b554-7a4284d83236 (at 10.8.1.7@o2ib6) reconnecting Jul 05 04:17:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 6be5edeb-cbb9-a4d7-5f1b-a3072b83c552 (at 10.8.1.7@o2ib6) Jul 05 04:17:03 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 04:21:41 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 04:21:41 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 309 previous similar messages Jul 05 04:31:52 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 04:31:52 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 219 previous similar messages Jul 05 04:36:29 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 05 04:36:29 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Jul 05 04:36:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 691e4f7c-24cc-f758-5354-96c1b01f1439 (at 10.8.7.7@o2ib6) reconnecting Jul 05 04:36:36 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 05 04:36:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 41d886bd-dfcd-3155-cafa-8df75781f2df (at 10.8.7.7@o2ib6) Jul 05 04:36:36 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 05 04:40:29 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 05 04:40:29 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Jul 05 04:40:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client e45eae18-7cf5-c24e-ada4-411d043e0647 (at 10.8.7.19@o2ib6) reconnecting Jul 05 04:40:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 6fb1a9aa-6234-c00b-63b2-a1a72639773f (at 10.8.7.19@o2ib6) Jul 05 04:41:54 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 04:41:54 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 351 previous similar messages Jul 05 04:52:06 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 04:52:06 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 266 previous similar messages Jul 05 04:57:38 fir-md1-s1 kernel: Lustre: 23605:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 05 05:02:10 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 05:02:10 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 343 previous similar messages Jul 05 05:12:11 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 05:12:11 fir-md1-s1 kernel: LustreError: 27583:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 330 previous similar messages Jul 05 05:22:19 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 05:22:19 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 297 previous similar messages Jul 05 05:32:20 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 05:32:20 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 308 previous similar messages Jul 05 05:42:24 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 05:42:24 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 314 previous similar messages Jul 05 05:52:39 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 05:52:39 fir-md1-s1 kernel: LustreError: 70067:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 321 previous similar messages Jul 05 06:02:42 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 05 06:02:42 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 313 previous similar messages Jul 05 06:12:57 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 06:12:57 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 319 previous similar messages Jul 05 06:22:59 fir-md1-s1 kernel: LustreError: 46591:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 05 06:22:59 fir-md1-s1 kernel: LustreError: 46591:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 335 previous similar messages Jul 05 06:33:08 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 06:33:08 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 262 previous similar messages Jul 05 06:39:29 fir-md1-s1 kernel: Lustre: 23634:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 05 06:43:11 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 06:43:11 fir-md1-s1 kernel: LustreError: 22428:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 303 previous similar messages Jul 05 06:53:16 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 06:53:16 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 279 previous similar messages Jul 05 07:03:20 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 07:03:20 fir-md1-s1 kernel: LustreError: 22958:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 257 previous similar messages Jul 05 07:04:54 fir-md1-s1 kernel: Lustre: 27320:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 05 07:13:24 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 07:13:24 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 260 previous similar messages Jul 05 07:23:24 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 07:23:24 fir-md1-s1 kernel: LustreError: 46521:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 295 previous similar messages Jul 05 07:33:24 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 05 07:33:24 fir-md1-s1 kernel: LustreError: 22990:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 371 previous similar messages Jul 05 07:43:29 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 05 07:43:29 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 326 previous similar messages Jul 05 07:49:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bec3d6e3-cbf4-befd-5ab3-86401c925d46 (at 10.9.0.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34ed637400, cur 1562338190 expire 1562338040 last 1562337963 Jul 05 07:49:50 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 07:49:50 fir-md1-s1 kernel: LustreError: 20384:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f0784263c00 x1636724756545744/t0(0) o104->fir-MDT0002@10.9.0.63@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 05 07:53:30 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 07:53:30 fir-md1-s1 kernel: LustreError: 21484:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 279 previous similar messages Jul 05 07:55:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c4e03c7e-ac09-b1ad-c42c-11e5ce21ec84 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1937e75800, cur 1562338532 expire 1562338382 last 1562338305 Jul 05 07:55:32 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 08:00:41 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 6726fecc-3078-ba4a-fb68-64e928250f1f (at 10.9.102.31@o2ib4) Jul 05 08:00:41 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 05 08:03:32 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 08:03:32 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 325 previous similar messages Jul 05 08:13:34 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 05 08:13:34 fir-md1-s1 kernel: LustreError: 22989:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 327 previous similar messages Jul 05 08:18:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 29e229ef-0b7d-e0ce-48dd-1c614dad7928 (at 10.9.112.15@o2ib4) Jul 05 08:18:27 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 08:19:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1cdcf44c-092e-67dd-29a2-3cb7e9bc7e29 (at 10.8.15.6@o2ib6) Jul 05 08:19:02 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 08:19:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to e79f4448-e890-1954-0996-0a25890d8ee5 (at 10.9.112.14@o2ib4) Jul 05 08:19:35 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 05 08:19:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1cdcf44c-092e-67dd-29a2-3cb7e9bc7e29 (at 10.8.15.6@o2ib6) Jul 05 08:19:56 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 08:20:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 2b30f37f-5bb9-7326-9800-1fc222ceb47c (at 10.9.106.61@o2ib4) Jul 05 08:21:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to df993956-2257-9a73-35ef-341b2f75d156 (at 10.9.106.58@o2ib4) Jul 05 08:21:26 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 08:22:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to cffa9ca6-4860-be91-20b9-abd21a031d37 (at 10.9.108.4@o2ib4) Jul 05 08:22:09 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 05 08:23:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ca62f9dd-676b-9343-5931-7cfc2e4cfe16 (at 10.9.0.63@o2ib4) Jul 05 08:23:24 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 05 08:23:38 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 08:23:38 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 305 previous similar messages Jul 05 08:25:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 8b986bcb-1a7e-3434-c2fb-c6a130bf7611 (at 10.9.104.25@o2ib4) Jul 05 08:25:21 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 08:33:43 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 08:33:43 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 355 previous similar messages Jul 05 08:41:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 7ab2f51d-a689-9f2c-be74-3bf003bf5840 (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f233c7fe800, cur 1562341269 expire 1562341119 last 1562341042 Jul 05 08:41:09 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 05 08:43:48 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 08:43:48 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 357 previous similar messages Jul 05 08:50:36 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client f2665e59-4b86-9898-62f9-cc1d6be44c9d (at 10.9.101.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3be27ca000, cur 1562341836 expire 1562341686 last 1562341609 Jul 05 08:50:36 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 08:50:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 03b80d11-11fc-47d1-78d0-c1090191edd3 (at 10.9.101.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2844712c00, cur 1562341842 expire 1562341692 last 1562341615 Jul 05 08:50:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 05 08:51:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02ea0e3d-c72b-2664-4a33-3841a13fb806 (at 10.9.101.55@o2ib4) Jul 05 08:51:02 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 05 08:53:59 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 08:53:59 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 298 previous similar messages Jul 05 09:04:01 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 09:04:01 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 270 previous similar messages Jul 05 09:14:09 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 09:14:09 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 335 previous similar messages Jul 05 09:17:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jul 05 09:17:24 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 09:24:10 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 09:24:10 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 270 previous similar messages Jul 05 09:34:12 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 09:34:12 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 227 previous similar messages Jul 05 09:40:02 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562344795/real 1562344795] req@ffff8f1a7b7d8300 x1636724902610976/t0(0) o106->fir-MDT0002@10.9.0.62@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562344802 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 05 09:40:02 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 05 09:40:10 fir-md1-s1 kernel: Lustre: 21446:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1a480fa400 x1634291378161312/t0(0) o101->9081d826-2f83-5b46-ff73-7e6473184838@10.8.17.25@o2ib6:15/0 lens 480/568 e 1 to 0 dl 1562344815 ref 2 fl Interpret:/0/0 rc 0/0 Jul 05 09:40:23 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562344816/real 1562344816] req@ffff8f1a7b7d8300 x1636724902610976/t0(0) o106->fir-MDT0002@10.9.0.62@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562344823 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 05 09:40:23 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 05 09:40:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 9081d826-2f83-5b46-ff73-7e6473184838 (at 10.8.17.25@o2ib6) reconnecting Jul 05 09:40:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 05 09:40:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 955a011e-49f5-8ef4-d629-f5f3f5327d18 (at 10.8.17.25@o2ib6) Jul 05 09:40:54 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 09:40:59 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562344851/real 1562344851] req@ffff8f1a7b7d8300 x1636724902610976/t0(0) o106->fir-MDT0002@10.9.0.62@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562344858 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 05 09:40:59 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 05 09:42:09 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562344922/real 1562344922] req@ffff8f1a7b7d8300 x1636724902610976/t0(0) o106->fir-MDT0002@10.9.0.62@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562344929 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 05 09:42:09 fir-md1-s1 kernel: Lustre: 20720:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 05 09:42:21 fir-md1-s1 kernel: Lustre: 50444:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f23715eb000 x1634186985277856/t0(0) o101->195f63e6-6435-e156-0d15-900ee8f39a3e@10.9.109.53@o2ib4:26/0 lens 480/568 e 1 to 0 dl 1562344946 ref 2 fl Interpret:/0/0 rc 0/0 Jul 05 09:42:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 195f63e6-6435-e156-0d15-900ee8f39a3e (at 10.9.109.53@o2ib4) reconnecting Jul 05 09:42:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 0c108c2d-a344-eca5-b660-99391625b78d (at 10.9.109.53@o2ib4) Jul 05 09:42:41 fir-md1-s1 kernel: LustreError: 50447:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.62@o2ib4) failed to reply to blocking AST (req@ffff8f1c47f2ec00 x1636724903797024 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f199ebafbc0/0x5d9ee631ae8958f4 lrc: 4/0,0 mode: PR/PR res: [0x2c002c05f:0xe02b:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x60000400000020 nid: 10.9.0.62@o2ib4 remote: 0x33d88ec1c734c2ba expref: 105515 pid: 23662 timeout: 1460043 lvb_type: 0 Jul 05 09:42:41 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.9.0.62@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 05 09:42:41 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.9.0.62@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f199ebafbc0/0x5d9ee631ae8958f4 lrc: 3/0,0 mode: PR/PR res: [0x2c002c05f:0xe02b:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x60000400000020 nid: 10.9.0.62@o2ib4 remote: 0x33d88ec1c734c2ba expref: 105516 pid: 23662 timeout: 0 lvb_type: 0 Jul 05 09:42:41 fir-md1-s1 kernel: Lustre: 20720:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (79:87s); client may timeout. req@ffff8f1a480fa400 x1634291378161312/t0(0) o101->9081d826-2f83-5b46-ff73-7e6473184838@10.8.17.25@o2ib6:15/0 lens 480/536 e 1 to 0 dl 1562344874 ref 1 fl Complete:/0/0 rc 301/301 Jul 05 09:42:41 fir-md1-s1 kernel: Lustre: 20720:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 20 previous similar messages Jul 05 09:42:58 fir-md1-s1 kernel: LustreError: 23737:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f3b4bffb300 x1636724904486656/t0(0) o104->fir-MDT0002@10.9.0.62@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 05 09:43:22 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148ebd6000, cur 1562345002 expire 1562344852 last 1562344775 Jul 05 09:43:23 fir-md1-s1 kernel: Lustre: 23630:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f3a5dcf7200 x1634186985352544/t0(0) o101->195f63e6-6435-e156-0d15-900ee8f39a3e@10.9.109.53@o2ib4:28/0 lens 480/568 e 0 to 0 dl 1562345008 ref 2 fl Interpret:/0/0 rc 0/0 Jul 05 09:43:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 3cd15e44-adf1-e977-3310-908c278e7f22 (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f39887c00, cur 1562345007 expire 1562344857 last 1562344780 Jul 05 09:43:27 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 05 09:43:27 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.9.0.62@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f195805d340/0x5d9ee631af012926 lrc: 3/0,0 mode: PR/PR res: [0x2c002c05f:0xe042:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x60000400000020 nid: 10.9.0.62@o2ib4 remote: 0x33d88ec1c745cfe2 expref: 7391 pid: 27316 timeout: 1460067 lvb_type: 0 Jul 05 09:43:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 195f63e6-6435-e156-0d15-900ee8f39a3e (at 10.9.109.53@o2ib4) reconnecting Jul 05 09:43:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 0c108c2d-a344-eca5-b660-99391625b78d (at 10.9.109.53@o2ib4) Jul 05 09:44:14 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 05 09:44:14 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 278 previous similar messages Jul 05 09:54:17 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 09:54:17 fir-md1-s1 kernel: LustreError: 46551:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 196 previous similar messages Jul 05 10:04:51 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 10:04:51 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 224 previous similar messages Jul 05 10:15:01 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 10:15:01 fir-md1-s1 kernel: LustreError: 20499:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 165 previous similar messages Jul 05 10:17:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jul 05 10:25:02 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 05 10:25:02 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 169 previous similar messages Jul 05 10:26:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jul 05 10:26:21 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 10:35:03 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 10:35:03 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 287 previous similar messages Jul 05 10:42:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bac6cd4e-a755-0f0e-da6d-e2c740eb12ce (at 10.9.114.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f05034fd800, cur 1562348573 expire 1562348423 last 1562348346 Jul 05 10:45:05 fir-md1-s1 kernel: LustreError: 21616:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 10:45:05 fir-md1-s1 kernel: LustreError: 21616:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 174 previous similar messages Jul 05 10:55:09 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 10:55:09 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 213 previous similar messages Jul 05 11:05:20 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 11:05:20 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 246 previous similar messages Jul 05 11:08:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 721f53d4-652b-e945-12ff-35ccdf15e929 (at 10.9.114.15@o2ib4) Jul 05 11:08:01 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 11:15:26 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 11:15:26 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 254 previous similar messages Jul 05 11:25:33 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 11:25:33 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 314 previous similar messages Jul 05 11:35:34 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 11:35:34 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 338 previous similar messages Jul 05 11:45:36 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 11:45:36 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 314 previous similar messages Jul 05 11:55:44 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 11:55:44 fir-md1-s1 kernel: LustreError: 46530:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 295 previous similar messages Jul 05 12:05:54 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 12:05:54 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 305 previous similar messages Jul 05 12:15:58 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 12:15:58 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 289 previous similar messages Jul 05 12:26:01 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 12:26:01 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 289 previous similar messages Jul 05 12:36:02 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 12:36:02 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 230 previous similar messages Jul 05 12:46:04 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 05 12:46:04 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 263 previous similar messages Jul 05 12:56:07 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 12:56:07 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 263 previous similar messages Jul 05 13:06:24 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 32768 GRANT, real grant 0 Jul 05 13:06:24 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 277 previous similar messages Jul 05 13:16:25 fir-md1-s1 kernel: LustreError: 81718:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 13:16:25 fir-md1-s1 kernel: LustreError: 81718:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 256 previous similar messages Jul 05 13:26:27 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 05 13:26:27 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 247 previous similar messages Jul 05 13:30:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9bcb994b-3f25-af85-c843-3a1243f52dea (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2097efec00, cur 1562358623 expire 1562358473 last 1562358396 Jul 05 13:30:23 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 13:30:38 fir-md1-s1 kernel: Lustre: MGS: Connection restored to acd26ab4-a020-fbc0-1a40-f0e7d759131f (at 10.8.23.14@o2ib6) Jul 05 13:30:38 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 13:36:32 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 13:36:32 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 238 previous similar messages Jul 05 13:46:52 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 13:46:52 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 186 previous similar messages Jul 05 13:56:56 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 13:56:56 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 224 previous similar messages Jul 05 14:07:09 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 14:07:09 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 264 previous similar messages Jul 05 14:17:15 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 14:17:15 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 233 previous similar messages Jul 05 14:25:47 fir-md1-s1 kernel: Lustre: 23571:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:25:48 fir-md1-s1 kernel: Lustre: 23554:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:25:50 fir-md1-s1 kernel: Lustre: 23554:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:25:53 fir-md1-s1 kernel: Lustre: 23649:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:25:53 fir-md1-s1 kernel: Lustre: 23649:0:(llog_cat.c:874:llog_cat_process_or_fork()) Skipped 1 previous similar message Jul 05 14:25:58 fir-md1-s1 kernel: Lustre: 23561:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:25:58 fir-md1-s1 kernel: Lustre: 23561:0:(llog_cat.c:874:llog_cat_process_or_fork()) Skipped 2 previous similar messages Jul 05 14:26:06 fir-md1-s1 kernel: Lustre: 23649:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:26:06 fir-md1-s1 kernel: Lustre: 23649:0:(llog_cat.c:874:llog_cat_process_or_fork()) Skipped 4 previous similar messages Jul 05 14:26:24 fir-md1-s1 kernel: Lustre: 23660:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:26:24 fir-md1-s1 kernel: Lustre: 23660:0:(llog_cat.c:874:llog_cat_process_or_fork()) Skipped 9 previous similar messages Jul 05 14:26:56 fir-md1-s1 kernel: Lustre: 21417:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:26:56 fir-md1-s1 kernel: Lustre: 21417:0:(llog_cat.c:874:llog_cat_process_or_fork()) Skipped 18 previous similar messages Jul 05 14:27:20 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 14:27:20 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 190 previous similar messages Jul 05 14:28:02 fir-md1-s1 kernel: Lustre: 23649:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:28:02 fir-md1-s1 kernel: Lustre: 23649:0:(llog_cat.c:874:llog_cat_process_or_fork()) Skipped 37 previous similar messages Jul 05 14:30:11 fir-md1-s1 kernel: Lustre: 21411:0:(llog_cat.c:874:llog_cat_process_or_fork()) fir-MDD0002: catlog [0x5:0xa:0x0] crosses index zero Jul 05 14:30:11 fir-md1-s1 kernel: Lustre: 21411:0:(llog_cat.c:874:llog_cat_process_or_fork()) Skipped 71 previous similar messages Jul 05 14:37:20 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 14:37:20 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 350 previous similar messages Jul 05 14:47:34 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 14:47:34 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 256 previous similar messages Jul 05 14:57:39 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 14:57:39 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 332 previous similar messages Jul 05 15:07:58 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 15:07:58 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 325 previous similar messages Jul 05 15:17:58 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 15:17:58 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 289 previous similar messages Jul 05 15:27:59 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 15:27:59 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 315 previous similar messages Jul 05 15:38:03 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 15:38:03 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 197 previous similar messages Jul 05 15:48:16 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 15:48:16 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 299 previous similar messages Jul 05 15:53:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 07f1d5f5-28d8-ec0b-6253-6164c1e142a5 (at 10.9.107.43@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14ee00c800, cur 1562367230 expire 1562367080 last 1562367003 Jul 05 15:53:50 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 15:54:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 07f1d5f5-28d8-ec0b-6253-6164c1e142a5 (at 10.9.107.43@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fc809c00, cur 1562367245 expire 1562367095 last 1562367018 Jul 05 15:58:17 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 15:58:17 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 300 previous similar messages Jul 05 16:08:33 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 16:08:33 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 221 previous similar messages Jul 05 16:12:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f15da91-4546-507e-8c99-9e08b5e219a4 (at 10.8.15.10@o2ib6) Jul 05 16:12:26 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 16:18:47 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 16:18:47 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 193 previous similar messages Jul 05 16:28:53 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 16:28:53 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 337 previous similar messages Jul 05 16:38:53 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 16:38:53 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 244 previous similar messages Jul 05 16:49:00 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 16:49:00 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 198 previous similar messages Jul 05 16:59:02 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 16:59:02 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 244 previous similar messages Jul 05 17:09:02 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 17:09:02 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 286 previous similar messages Jul 05 17:19:14 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 17:19:14 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 153 previous similar messages Jul 05 17:29:15 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 17:29:15 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 282 previous similar messages Jul 05 17:39:43 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 17:39:43 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 322 previous similar messages Jul 05 17:49:43 fir-md1-s1 kernel: LustreError: 21617:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 17:49:43 fir-md1-s1 kernel: LustreError: 21617:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 325 previous similar messages Jul 05 17:59:45 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 17:59:45 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 309 previous similar messages Jul 05 18:09:46 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 18:09:46 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 345 previous similar messages Jul 05 18:20:33 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 18:20:33 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 314 previous similar messages Jul 05 18:30:33 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 18:30:33 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 270 previous similar messages Jul 05 18:40:43 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 05 18:40:43 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 262 previous similar messages Jul 05 18:50:47 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 18:50:47 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 221 previous similar messages Jul 05 19:00:48 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 05 19:00:48 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 291 previous similar messages Jul 05 19:11:11 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 19:11:11 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 225 previous similar messages Jul 05 19:21:13 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 19:21:13 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 216 previous similar messages Jul 05 19:31:15 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 19:31:15 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 220 previous similar messages Jul 05 19:36:16 fir-md1-s1 kernel: Lustre: 97656:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562380569/real 1562380569] req@ffff8f1f19f2ec00 x1636725341747536/t0(0) o104->fir-MDT0000@10.8.0.66@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562380576 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 05 19:36:16 fir-md1-s1 kernel: Lustre: 97656:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 05 19:36:24 fir-md1-s1 kernel: Lustre: 97638:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2229669200 x1631551722727968/t412995623177(0) o36->78ab2c22-394d-bdd4-0b8e-3553d6a47e28@10.8.17.2@o2ib6:29/0 lens 488/3152 e 1 to 0 dl 1562380589 ref 2 fl Interpret:/0/0 rc 0/0 Jul 05 19:36:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 78ab2c22-394d-bdd4-0b8e-3553d6a47e28 (at 10.8.17.2@o2ib6) reconnecting Jul 05 19:36:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9b9b8332-39fb-197d-4c4c-38d36ae981cd (at 10.8.17.2@o2ib6) Jul 05 19:36:30 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 19:36:36 fir-md1-s1 kernel: Lustre: 23702:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562380589/real 1562380589] req@ffff8f41cca51b00 x1636725341976576/t0(0) o106->fir-MDT0000@10.8.0.66@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562380596 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 05 19:36:36 fir-md1-s1 kernel: Lustre: 23702:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 05 19:36:44 fir-md1-s1 kernel: LustreError: 97656:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.0.66@o2ib6) failed to reply to blocking AST (req@ffff8f1f19f2ec00 x1636725341747536 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f17a7279440/0x5d9ee6328c624984 lrc: 4/0,0 mode: PR/PR res: [0x200029c29:0x17d:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.0.66@o2ib6 remote: 0xffcd23c129cbc86f expref: 2404 pid: 21434 timeout: 1495686 lvb_type: 0 Jul 05 19:36:44 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.0.66@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 05 19:36:44 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.0.66@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f17a7279440/0x5d9ee6328c624984 lrc: 3/0,0 mode: PR/PR res: [0x200029c29:0x17d:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.0.66@o2ib6 remote: 0xffcd23c129cbc86f expref: 2405 pid: 21434 timeout: 0 lvb_type: 0 Jul 05 19:36:44 fir-md1-s1 kernel: LustreError: 24582:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1fbe695d00 x1636725342147392/t0(0) o104->fir-MDT0000@10.8.0.66@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 05 19:39:39 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 08dbb8a3-6486-471a-a832-58e0c151a878 (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2438cf0400, cur 1562380779 expire 1562380629 last 1562380552 Jul 05 19:39:39 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 05 19:41:15 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 19:41:15 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 247 previous similar messages Jul 05 19:51:23 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 19:51:23 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 200 previous similar messages Jul 05 20:01:33 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 20:01:33 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 221 previous similar messages Jul 05 20:03:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 048975c6-ab6c-4dc2-089d-bee623fa3e4d (at 10.9.114.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2505b8d000, cur 1562382211 expire 1562382061 last 1562381984 Jul 05 20:03:31 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 05 20:03:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 048975c6-ab6c-4dc2-089d-bee623fa3e4d (at 10.9.114.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1477697c00, cur 1562382212 expire 1562382062 last 1562381985 Jul 05 20:03:32 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 05 20:11:35 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 20:11:35 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 228 previous similar messages Jul 05 20:21:40 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 20:21:40 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 274 previous similar messages Jul 05 20:28:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.113.10@o2ib4) Jul 05 20:29:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to beb38144-d000-b47c-bba7-ccce9e6df4a5 (at 10.9.114.10@o2ib4) Jul 05 20:29:07 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 20:31:41 fir-md1-s1 kernel: LustreError: 21617:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 05 20:31:41 fir-md1-s1 kernel: LustreError: 21617:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 235 previous similar messages Jul 05 20:41:43 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 20:41:43 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 350 previous similar messages Jul 05 20:51:46 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 20:51:46 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 286 previous similar messages Jul 05 21:01:47 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 21:01:47 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 246 previous similar messages Jul 05 21:11:53 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 21:11:53 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 334 previous similar messages Jul 05 21:22:06 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 21:22:06 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 307 previous similar messages Jul 05 21:32:16 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 21:32:16 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 228 previous similar messages Jul 05 21:42:17 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 21:42:17 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 273 previous similar messages Jul 05 21:51:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client cc538e45-b702-a36c-5f06-e62f44bf19d0 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f450665fc00, cur 1562388698 expire 1562388548 last 1562388471 Jul 05 21:51:38 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 05 21:52:20 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 05 21:52:20 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 265 previous similar messages Jul 05 22:02:23 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 22:02:23 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 293 previous similar messages Jul 05 22:12:29 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 22:12:29 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 165 previous similar messages Jul 05 22:22:35 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 22:22:35 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 251 previous similar messages Jul 05 22:32:36 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 22:32:36 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 276 previous similar messages Jul 05 22:42:36 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 22:42:36 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 283 previous similar messages Jul 05 22:52:45 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 22:52:45 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 230 previous similar messages Jul 05 23:02:47 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 23:02:47 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 287 previous similar messages Jul 05 23:12:50 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 23:12:50 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 241 previous similar messages Jul 05 23:22:54 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 23:22:54 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 360 previous similar messages Jul 05 23:32:59 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 23:32:59 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 261 previous similar messages Jul 05 23:43:00 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 05 23:43:00 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 311 previous similar messages Jul 05 23:49:47 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562395780/real 1562395780] req@ffff8f12b6453000 x1636725417782288/t0(0) o106->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562395787 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 05 23:49:47 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 05 23:49:54 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562395787/real 1562395787] req@ffff8f12b6453000 x1636725417782288/t0(0) o106->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562395794 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 05 23:49:55 fir-md1-s1 kernel: Lustre: 21417:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0a72f19e00 x1637014291926288/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:0/0 lens 520/568 e 1 to 0 dl 1562395800 ref 2 fl Interpret:/0/0 rc 0/0 Jul 05 23:50:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jul 05 23:50:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jul 05 23:50:01 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 23:50:08 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562395801/real 1562395801] req@ffff8f12b6453000 x1636725417782288/t0(0) o106->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562395808 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 05 23:50:08 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 05 23:50:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jul 05 23:50:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jul 05 23:50:29 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562395822/real 1562395822] req@ffff8f12b6453000 x1636725417782288/t0(0) o106->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562395829 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 05 23:50:29 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 05 23:50:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jul 05 23:50:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jul 05 23:51:04 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562395857/real 1562395857] req@ffff8f12b6453000 x1636725417782288/t0(0) o106->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562395864 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 05 23:51:04 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 05 23:51:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jul 05 23:51:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jul 05 23:51:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jul 05 23:51:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jul 05 23:51:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jul 05 23:51:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jul 05 23:52:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6ee172d9-72a9-7fa2-230d-3850214207fa (at 10.0.10.3@o2ib7) reconnecting Jul 05 23:52:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 72ddd52f-2877-4d72-483b-2a30690dc155 (at 10.0.10.3@o2ib7) Jul 05 23:52:14 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562395927/real 1562395927] req@ffff8f12b6453000 x1636725417782288/t0(0) o106->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562395934 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 05 23:52:14 fir-md1-s1 kernel: Lustre: 23660:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 05 23:52:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 579568b0-fc84-54e9-66a5-a75bc316659b (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f22e81e0000, cur 1562395943 expire 1562395793 last 1562395716 Jul 05 23:52:23 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 05 23:53:07 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 05 23:53:07 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 315 previous similar messages Jul 06 00:03:17 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 00:03:17 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 301 previous similar messages Jul 06 00:13:19 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 00:13:19 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 248 previous similar messages Jul 06 00:23:23 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 06 00:23:23 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 231 previous similar messages Jul 06 00:33:30 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 00:33:30 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 227 previous similar messages Jul 06 00:43:59 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 00:43:59 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 279 previous similar messages Jul 06 00:54:00 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 00:54:00 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 213 previous similar messages Jul 06 01:04:01 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 01:04:01 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 225 previous similar messages Jul 06 01:14:19 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 01:14:19 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 216 previous similar messages Jul 06 01:24:22 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 01:24:22 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 306 previous similar messages Jul 06 01:34:29 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 01:34:29 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 248 previous similar messages Jul 06 01:44:30 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 01:44:30 fir-md1-s1 kernel: LustreError: 27605:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 275 previous similar messages Jul 06 01:54:47 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 01:54:47 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 218 previous similar messages Jul 06 02:04:50 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 02:04:50 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 294 previous similar messages Jul 06 02:14:54 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 06 02:14:54 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 279 previous similar messages Jul 06 02:25:21 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 02:25:21 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 350 previous similar messages Jul 06 02:35:22 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 02:35:22 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 335 previous similar messages Jul 06 02:45:30 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 02:45:30 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 333 previous similar messages Jul 06 02:55:37 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 02:55:37 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 277 previous similar messages Jul 06 03:06:06 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 03:06:06 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 291 previous similar messages Jul 06 03:16:07 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 03:16:07 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 289 previous similar messages Jul 06 03:26:31 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 06 03:26:31 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 248 previous similar messages Jul 06 03:36:35 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 03:36:35 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 282 previous similar messages Jul 06 03:46:38 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 03:46:38 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 254 previous similar messages Jul 06 03:56:40 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 03:56:40 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 303 previous similar messages Jul 06 04:06:59 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 04:06:59 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 272 previous similar messages Jul 06 04:15:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d7baa7ce-5705-6e23-2846-5c2b64fab1c8 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f172772fc00, cur 1562411703 expire 1562411553 last 1562411476 Jul 06 04:15:03 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 04:15:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jul 06 04:17:02 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 06 04:17:02 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 218 previous similar messages Jul 06 04:27:34 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 04:27:34 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 239 previous similar messages Jul 06 04:37:40 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 06 04:37:40 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 245 previous similar messages Jul 06 04:47:41 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 04:47:41 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 170 previous similar messages Jul 06 04:57:46 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 04:57:46 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 219 previous similar messages Jul 06 05:07:47 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 05:07:47 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 254 previous similar messages Jul 06 05:17:52 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 05:17:52 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 376 previous similar messages Jul 06 05:28:00 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 05:28:00 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 204 previous similar messages Jul 06 05:38:17 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 05:38:17 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 368 previous similar messages Jul 06 05:48:20 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 05:48:20 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 353 previous similar messages Jul 06 05:58:21 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 05:58:21 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 260 previous similar messages Jul 06 06:08:22 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 06:08:22 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 290 previous similar messages Jul 06 06:18:26 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 06:18:26 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 319 previous similar messages Jul 06 06:28:32 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 06:28:32 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 287 previous similar messages Jul 06 06:38:33 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 06:38:33 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 268 previous similar messages Jul 06 06:48:43 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 06:48:43 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 256 previous similar messages Jul 06 06:58:46 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 06 06:58:46 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 268 previous similar messages Jul 06 07:08:55 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 06 07:08:55 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 256 previous similar messages Jul 06 07:19:04 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 07:19:04 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 213 previous similar messages Jul 06 07:29:19 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 06 07:29:19 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 239 previous similar messages Jul 06 07:39:24 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 06 07:39:24 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 237 previous similar messages Jul 06 07:49:49 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 32768 GRANT, real grant 0 Jul 06 07:49:49 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 236 previous similar messages Jul 06 07:58:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1cdcf44c-092e-67dd-29a2-3cb7e9bc7e29 (at 10.8.15.6@o2ib6) Jul 06 07:58:51 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 08:00:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ea8d1cad-7733-1759-3045-271c39c8bfa7 (at 10.9.114.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3a8b77e400, cur 1562425246 expire 1562425096 last 1562425019 Jul 06 08:00:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 08:01:06 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 08:01:06 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 284 previous similar messages Jul 06 08:11:13 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 08:11:13 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 319 previous similar messages Jul 06 08:22:05 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 08:22:05 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 263 previous similar messages Jul 06 08:22:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9efc11a2-2302-21f2-1382-b7d75650f9a7 (at 10.9.113.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f13b5844800, cur 1562426534 expire 1562426384 last 1562426307 Jul 06 08:22:14 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 08:27:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to beb38144-d000-b47c-bba7-ccce9e6df4a5 (at 10.9.114.10@o2ib4) Jul 06 08:27:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 08:32:05 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 08:32:05 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 279 previous similar messages Jul 06 08:42:11 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 08:42:11 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 269 previous similar messages Jul 06 08:47:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.113.10@o2ib4) Jul 06 08:47:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 08:53:24 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 08:53:24 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 326 previous similar messages Jul 06 09:03:25 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 09:03:25 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 244 previous similar messages Jul 06 09:04:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 810ae33a-f2a4-73ad-b573-a8509a545499 (at 10.8.0.66@o2ib6) Jul 06 09:04:18 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 09:08:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d6c51075-12c4-bfee-f317-56a8e3a97c90 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1e48a01000, cur 1562429294 expire 1562429144 last 1562429067 Jul 06 09:08:14 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 09:08:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jul 06 09:08:28 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 09:13:25 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 09:13:25 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 232 previous similar messages Jul 06 09:23:28 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 94208 GRANT, real grant 0 Jul 06 09:23:28 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 357 previous similar messages Jul 06 09:33:49 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 09:33:49 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 212 previous similar messages Jul 06 09:44:10 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 09:44:10 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 176 previous similar messages Jul 06 09:54:14 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 09:54:14 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 204 previous similar messages Jul 06 10:04:16 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 10:04:16 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 286 previous similar messages Jul 06 10:14:58 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 10:14:58 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 196 previous similar messages Jul 06 10:25:46 fir-md1-s1 kernel: LustreError: 21617:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 10:25:46 fir-md1-s1 kernel: LustreError: 21617:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 221 previous similar messages Jul 06 10:35:50 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 10:35:50 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 254 previous similar messages Jul 06 10:46:34 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 10:46:34 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 232 previous similar messages Jul 06 10:56:42 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 10:56:42 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 260 previous similar messages Jul 06 11:06:42 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 11:06:42 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 179 previous similar messages Jul 06 11:16:50 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 11:16:50 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 264 previous similar messages Jul 06 11:27:14 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 11:27:14 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 243 previous similar messages Jul 06 11:37:22 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 11:37:22 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 207 previous similar messages Jul 06 11:47:25 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 11:47:25 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 191 previous similar messages Jul 06 11:57:50 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 11:57:50 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 191 previous similar messages Jul 06 12:08:35 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 12:08:35 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 161 previous similar messages Jul 06 12:18:39 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 12:18:39 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 237 previous similar messages Jul 06 12:28:57 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 12:28:57 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 110 previous similar messages Jul 06 12:38:57 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 12:38:57 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 98 previous similar messages Jul 06 12:49:12 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 12:49:12 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 131 previous similar messages Jul 06 13:00:10 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 06 13:00:10 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 74 previous similar messages Jul 06 13:10:13 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 06 13:10:13 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 74 previous similar messages Jul 06 13:20:52 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 06 13:20:52 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 120 previous similar messages Jul 06 13:31:02 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 13:31:02 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 67 previous similar messages Jul 06 13:41:07 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 06 13:41:07 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 58 previous similar messages Jul 06 13:55:11 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 06 13:55:11 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 36 previous similar messages Jul 06 14:06:37 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 06 14:06:37 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22 previous similar messages Jul 06 14:19:59 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 32768 GRANT, real grant 0 Jul 06 14:19:59 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 06 14:31:12 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 06 14:31:12 fir-md1-s1 kernel: LustreError: 42895:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 06 14:46:42 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 14:46:42 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 06 14:57:09 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 14:57:09 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 25 previous similar messages Jul 06 15:07:44 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 15:07:44 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 06 15:34:05 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 15:34:05 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 15:38:35 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 06 15:38:35 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 06 15:43:15 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 15:43:15 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 06 15:49:20 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 15:49:20 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 06 16:00:03 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 16:00:03 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 06 16:10:43 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 16:10:43 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 21 previous similar messages Jul 06 16:20:47 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 16:20:47 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 16:30:50 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 16:30:50 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jul 06 16:41:45 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 16:41:45 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 16:51:53 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 16:51:53 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 06 17:01:54 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 17:01:54 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 06 17:12:18 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 17:12:18 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 10 previous similar messages Jul 06 17:23:13 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 17:23:13 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 06 17:33:30 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 17:33:30 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jul 06 17:43:42 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 17:43:42 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jul 06 17:53:49 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 17:53:49 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 06 18:04:37 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 18:04:37 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 06 18:14:48 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 18:14:48 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 18:25:03 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 18:25:03 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 06 18:35:48 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 18:35:48 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 18:44:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 43d4db84-f4df-a5c1-f438-2ed5ad3ddb7d (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2429a3e800, cur 1562463857 expire 1562463707 last 1562463630 Jul 06 18:44:17 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 18:44:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00304ea7-578d-2727-24ce-d8f8efb87890 (at 10.8.26.4@o2ib6) Jul 06 18:44:58 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 06 18:46:23 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 18:46:23 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 06 18:57:07 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 18:57:07 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 06 19:07:28 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 19:07:28 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 06 19:17:53 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 19:17:53 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 10 previous similar messages Jul 06 19:28:28 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 19:28:28 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 19:38:52 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 19:38:52 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 06 19:49:16 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 19:49:16 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 06 19:59:35 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 19:59:35 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 06 20:10:08 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 20:10:08 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 06 20:20:41 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 20:20:41 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 20:31:15 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 20:31:15 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 06 20:42:23 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 20:42:23 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 20:52:40 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 20:52:40 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 06 21:03:01 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 21:03:01 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 06 21:13:58 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 21:13:58 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jul 06 21:25:11 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 21:25:11 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jul 06 21:35:34 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 21:35:34 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jul 06 21:46:48 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 21:46:48 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 06 21:57:13 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 21:57:13 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 9 previous similar messages Jul 06 22:07:16 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 22:07:16 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 06 22:17:40 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 22:17:40 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 06 22:28:29 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 22:28:29 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jul 06 22:39:05 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 22:39:05 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 06 22:50:10 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 22:50:10 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 06 23:00:45 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 23:00:45 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jul 06 23:10:59 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 23:10:59 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 06 23:21:44 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 23:21:44 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 06 23:32:14 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 23:32:14 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 06 23:42:35 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 23:42:35 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 11 previous similar messages Jul 06 23:53:21 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 06 23:53:21 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 15 previous similar messages Jul 07 00:03:41 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 00:03:41 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jul 07 00:14:01 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 00:14:01 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 07 00:24:21 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 00:24:21 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jul 07 00:35:14 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 00:35:14 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 19 previous similar messages Jul 07 00:46:06 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 00:46:06 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 18 previous similar messages Jul 07 00:56:10 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 00:56:10 fir-md1-s1 kernel: LustreError: 20503:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 21 previous similar messages Jul 07 01:18:25 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 01:18:25 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 10 previous similar messages Jul 07 01:20:42 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 01:20:42 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 01:32:20 fir-md1-s1 kernel: LustreError: 22059:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 01:46:29 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 02:20:09 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 02:20:09 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 93 previous similar messages Jul 07 02:21:36 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 02:32:04 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 02:41:01 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 02:41:01 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 03:17:44 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 03:25:04 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 03:40:19 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 03:44:04 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 03:44:46 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 07 03:44:46 fir-md1-s1 kernel: LustreError: 46526:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 31 previous similar messages Jul 07 03:46:37 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 07 03:46:37 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 20 previous similar messages Jul 07 03:50:25 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 07 03:50:25 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 04:39:04 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 07 04:39:04 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 06:56:47 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 07:09:12 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:09:42 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:09:47 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:09:47 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 07:10:17 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:10:17 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 07:10:22 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:11:02 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:11:02 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 07:11:32 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:11:32 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 07 07:12:35 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:12:35 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 07 07:14:21 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:14:21 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 07 07:17:17 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:17:17 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 07 07:22:32 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:22:32 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 07 07:32:45 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:32:45 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 41 previous similar messages Jul 07 07:43:04 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:43:04 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 50 previous similar messages Jul 07 07:53:06 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 07:53:06 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 44 previous similar messages Jul 07 08:03:23 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 08:03:23 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 50 previous similar messages Jul 07 08:13:40 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 08:13:40 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 48 previous similar messages Jul 07 08:23:58 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 08:23:58 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 44 previous similar messages Jul 07 08:34:16 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 08:34:16 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 43 previous similar messages Jul 07 08:44:24 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 08:44:24 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 45 previous similar messages Jul 07 08:54:52 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 08:54:52 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 47 previous similar messages Jul 07 09:05:09 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 09:05:09 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 43 previous similar messages Jul 07 09:15:20 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 07 09:15:20 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 41 previous similar messages Jul 07 09:25:49 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 07 09:25:49 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 44 previous similar messages Jul 07 09:35:51 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 09:35:51 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 45 previous similar messages Jul 07 09:46:08 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 09:46:08 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 44 previous similar messages Jul 07 09:56:35 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 09:56:35 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 52 previous similar messages Jul 07 10:06:39 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 10:06:39 fir-md1-s1 kernel: LustreError: 21389:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 45 previous similar messages Jul 07 10:16:56 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 10:16:56 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 49 previous similar messages Jul 07 10:26:57 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 10:26:57 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 43 previous similar messages Jul 07 10:37:14 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 10:37:14 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 45 previous similar messages Jul 07 10:47:21 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 10:47:21 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 48 previous similar messages Jul 07 10:57:28 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 07 10:57:28 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 33 previous similar messages Jul 07 11:26:39 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 11:26:39 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 07 11:53:26 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 11:58:47 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 11:58:47 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 12:04:29 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 12:07:51 fir-md1-s1 kernel: LustreError: 22649:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 12:11:33 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 12:21:55 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 13:06:31 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 13:08:04 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 13:15:26 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 13:21:32 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 13:32:23 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 13:48:00 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 13:48:00 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 07 14:22:30 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 14:22:30 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 14:24:16 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 14:30:51 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 14:55:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3903bb43-6d23-19dc-ccc3-5eecafcff35a (at 10.8.1.36@o2ib6) reconnecting Jul 07 14:55:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 6cf8bc2f-bf0f-5ecb-1a1d-10eb0db43353 (at 10.8.1.36@o2ib6) Jul 07 14:55:23 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 14:56:45 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 15:01:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 15:01:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 15:01:33 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 07 15:01:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 15:12:42 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 15:23:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 15:23:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 15:23:24 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 15:23:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 15:39:46 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 07 15:48:19 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 16:29:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 26320709-561f-90ed-6684-fea46854b319 (at 10.8.1.29@o2ib6) Jul 07 16:39:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 16:39:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 16:39:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 17:01:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 17:01:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 17:07:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 17:07:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 17:08:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 07 17:08:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 17:08:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 17:20:34 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 17:21:31 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 17:50:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 17:50:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 17:50:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 17:50:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 17:56:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 17:56:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 18:02:17 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:02:20 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 18:02:27 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:02:40 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:02:54 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:03:11 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 18:03:30 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:03:30 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 18:04:11 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:04:11 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jul 07 18:05:22 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:05:22 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 8 previous similar messages Jul 07 18:07:36 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:07:36 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 16 previous similar messages Jul 07 18:11:53 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:11:53 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 29 previous similar messages Jul 07 18:20:31 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:20:31 fir-md1-s1 kernel: LustreError: 27481:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 65 previous similar messages Jul 07 18:30:33 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 18:30:33 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 77 previous similar messages Jul 07 18:40:41 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 155648 GRANT, real grant 0 Jul 07 18:40:41 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 72 previous similar messages Jul 07 19:08:47 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 32768 GRANT, real grant 0 Jul 07 19:08:47 fir-md1-s1 kernel: LustreError: 46576:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 54 previous similar messages Jul 07 19:52:37 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 07 20:32:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 20:32:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 20:33:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 617d800a-afeb-08ed-bb4c-9f77025769ad (at 10.8.25.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2507968000, cur 1562556788 expire 1562556638 last 1562556561 Jul 07 20:33:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 20:34:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 374fd2d9-2972-20b7-dfa4-bf6b2470cf36 (at 10.8.1.6@o2ib6) reconnecting Jul 07 20:34:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.1.6@o2ib6, removing former export from same NID Jul 07 20:34:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 263eaecf-e81f-64c0-76c4-67b409a3186f (at 10.8.1.6@o2ib6) Jul 07 20:37:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 20:37:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 20:37:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 20:37:35 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 20:38:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 20:38:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 20:38:25 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 20:41:29 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 07 20:41:29 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 20:47:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 20:47:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 21:07:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 21:07:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 21:23:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 0d321477-e1a4-6634-93cf-b59d753ff98f (at 10.8.18.6@o2ib6) reconnecting Jul 07 21:23:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b1f9ccc8-925b-b4d2-9293-aac9aa183623 (at 10.8.18.6@o2ib6) Jul 07 21:40:22 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 21:40:27 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 21:40:27 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 21:40:28 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 21:40:28 fir-md1-s1 kernel: LustreError: 46532:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 07 21:40:30 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 21:40:30 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 07 21:40:35 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 21:40:35 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 07 21:40:56 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 07 21:40:56 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jul 07 21:42:00 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 21:42:00 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 26 previous similar messages Jul 07 21:42:57 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 07 21:42:57 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 32 previous similar messages Jul 07 21:44:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 730be893-31e8-983c-06e1-f426e82a434b (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f20e2733400, cur 1562561065 expire 1562560915 last 1562560838 Jul 07 21:44:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 21:44:33 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 21:44:33 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 13 previous similar messages Jul 07 21:46:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 26320709-561f-90ed-6684-fea46854b319 (at 10.8.1.29@o2ib6) Jul 07 21:54:15 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 07 21:54:15 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 22:21:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.1.17@o2ib6, removing former export from same NID Jul 07 22:21:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 7f5b8d8c-996c-1887-f76d-12c3566ba896 (at 10.8.1.17@o2ib6) reconnecting Jul 07 22:21:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to caccb606-7559-916b-0433-b661c183f103 (at 10.8.1.17@o2ib6) Jul 07 22:21:41 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 22:21:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1d02240d-6817-4f2d-eb33-71d0a2e61934 (at 10.8.18.3@o2ib6) reconnecting Jul 07 22:21:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 837c124c-41d9-368d-aae3-f10235137c33 (at 10.8.18.3@o2ib6) Jul 07 22:21:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 22:21:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.18.3@o2ib6, removing former export from same NID Jul 07 22:22:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.18.3@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 07 22:22:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1d02240d-6817-4f2d-eb33-71d0a2e61934 (at 10.8.18.3@o2ib6) reconnecting Jul 07 22:22:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 837c124c-41d9-368d-aae3-f10235137c33 (at 10.8.18.3@o2ib6) Jul 07 22:22:35 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 22:22:35 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 22:33:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 22:33:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 22:33:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 22:33:44 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 22:34:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 22:34:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 22:34:31 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 22:47:24 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 22:55:05 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 23:00:36 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 23:00:36 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 23:08:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:08:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:08:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:09:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:09:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:09:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:09:13 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:16:57 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 07 23:18:11 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 23:21:11 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 23:22:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:22:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:22:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:22:06 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:22:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 07 23:22:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:22:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:22:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:22:55 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:23:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:23:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:23:03 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:26:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:26:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:27:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:27:32 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:31:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 9eb449c2-e54f-1e34-81bc-f024b214ecc1 (at 10.9.114.3@o2ib4) reconnecting Jul 07 23:31:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to bff84d1e-0a69-b6c4-379f-b22c9974d598 (at 10.9.114.3@o2ib4) Jul 07 23:33:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:33:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:33:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:33:29 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:34:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:34:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:34:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:34:23 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 23:35:19 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 23:35:19 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 23:36:46 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 23:37:30 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 07 23:37:30 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 07 23:38:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:38:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:38:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:38:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:38:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 07 23:38:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 07 23:38:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:38:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:38:37 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:38:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 07 23:51:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:51:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:51:44 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:51:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:51:44 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jul 07 23:51:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 07 23:52:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 07 23:52:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 07 23:52:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 07 23:52:16 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 07 23:52:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 00:06:43 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 00:06:43 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 08 00:14:34 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 00:25:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 0d321477-e1a4-6634-93cf-b59d753ff98f (at 10.8.18.6@o2ib6) reconnecting Jul 08 00:25:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b1f9ccc8-925b-b4d2-9293-aac9aa183623 (at 10.8.18.6@o2ib6) Jul 08 00:25:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 00:40:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 00:40:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 00:52:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.1.6@o2ib6, removing former export from same NID Jul 08 00:52:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 374fd2d9-2972-20b7-dfa4-bf6b2470cf36 (at 10.8.1.6@o2ib6) reconnecting Jul 08 00:52:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 263eaecf-e81f-64c0-76c4-67b409a3186f (at 10.8.1.6@o2ib6) Jul 08 00:53:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.1.6@o2ib6, removing former export from same NID Jul 08 00:53:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 263eaecf-e81f-64c0-76c4-67b409a3186f (at 10.8.1.6@o2ib6) Jul 08 00:53:02 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 01:03:22 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 08 01:07:25 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 01:07:25 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 08 01:07:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 01:07:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 01:07:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 01:07:58 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 01:08:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 01:08:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 01:08:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 01:08:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 01:08:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 01:08:43 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 01:08:43 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 01:09:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 01:09:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 01:09:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 01:09:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 01:09:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 01:09:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 01:09:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 01:09:29 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 01:09:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 01:09:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 01:09:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 01:10:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 01:10:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 01:10:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 01:10:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 01:13:02 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 01:13:15 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 08 01:18:58 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 01:22:23 fir-md1-s1 kernel: LustreError: 46581:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 01:25:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 01:25:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 01:25:46 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 01:29:17 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 01:30:53 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 01:36:29 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 01:36:29 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 4 previous similar messages Jul 08 02:01:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 02:01:49 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 02:03:44 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 02:12:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 02:12:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 02:14:22 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 02:14:22 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 08 02:19:39 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 02:26:26 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 02:26:26 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 08 02:27:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 02:27:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 02:27:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 02:27:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 02:28:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 02:28:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 02:28:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 02:28:18 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 02:28:18 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 02:49:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 02:49:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 02:49:31 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 03:23:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 03:23:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 03:23:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 03:24:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 03:24:08 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 03:24:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 03:24:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 03:24:32 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 03:24:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 03:24:32 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 03:26:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 03:26:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 03:26:43 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 03:29:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 03:29:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 03:29:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 03:29:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 03:29:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 03:38:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 03:38:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 03:38:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 03:38:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 03:38:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 03:46:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 03:46:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 03:46:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 03:46:11 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 03:46:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 04:22:59 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 08 04:22:59 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 08 04:26:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 04:26:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 04:26:51 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 08 04:26:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 04:26:51 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 04:27:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 04:27:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 04:27:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 04:27:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 04:27:33 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 04:28:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 04:28:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 04:28:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 04:28:33 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 04:29:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 04:29:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 04:32:48 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 08 04:32:48 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 08 04:33:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 04:33:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 04:39:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 04:39:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 04:39:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 04:39:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 04:39:33 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 05:15:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 05:15:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 05:15:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 05:15:42 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 08 05:15:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 05:15:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 05:15:59 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 05:16:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 05:16:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 05:16:24 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 05:16:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 05:16:24 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 08 05:17:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 05:17:21 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 05:17:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 05:17:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 05:17:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 05:17:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 05:17:46 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 08 05:18:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 05:18:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 05:18:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 05:18:30 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 05:20:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 05:20:07 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 08 05:20:38 fir-md1-s1 kernel: LustreError: 55546:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8f1f53e44850 x1631562238067040/t0(0) o256->9d0e62c0-e368-6db8-c860-d1e71d1366bc@10.8.17.11@o2ib6:13/0 lens 304/240 e 0 to 0 dl 1562588443 ref 1 fl Interpret:/0/0 rc 0/0 Jul 08 05:20:38 fir-md1-s1 kernel: LustreError: 55546:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 6 previous similar messages Jul 08 05:21:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 05:21:02 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 08 05:29:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) reconnecting Jul 08 05:29:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 9d0e62c0-e368-6db8-c860-d1e71d1366bc (at 10.8.17.11@o2ib6) Jul 08 05:29:43 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 05:29:43 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 05:29:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 07:42:13 fir-md1-s1 kernel: LustreError: 21535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 08 07:52:02 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 155648 GRANT, real grant 0 Jul 08 07:52:02 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 08 08:10:13 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 08 08:19:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 26320709-561f-90ed-6684-fea46854b319 (at 10.8.1.29@o2ib6) Jul 08 08:19:08 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 08:48:16 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 08 08:48:51 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli c9c3f7fc-2b8d-1a18-fd16-3c9107a89baf claims 28672 GRANT, real grant 0 Jul 08 08:55:27 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 08 08:55:27 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 08 09:38:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 374fd2d9-2972-20b7-dfa4-bf6b2470cf36 (at 10.8.1.6@o2ib6) reconnecting Jul 08 09:38:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 263eaecf-e81f-64c0-76c4-67b409a3186f (at 10.8.1.6@o2ib6) Jul 08 09:38:55 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 09:41:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 08 09:41:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 08 09:41:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 09:41:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 08 09:42:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:42:14 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 09:42:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 09:42:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 09:42:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 09:42:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 09:43:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 09:43:31 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 09:44:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:44:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 09:44:24 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 09:44:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 08 09:44:24 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 08 09:44:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 09:44:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:45:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:45:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 09:45:48 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 08 09:46:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 09:46:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:46:27 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 09:46:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 09:46:52 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 09:46:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 09:46:52 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 08 09:47:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.26.33@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:47:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.26.35@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:48:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 09:48:01 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 09:48:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 09:48:01 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 08 09:49:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.19@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:50:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 09:50:50 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 08 09:50:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.67@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:50:51 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 08 09:51:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 08 09:51:34 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 08 09:53:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.20.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:53:02 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 08 09:53:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 09:53:18 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jul 08 09:55:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 09:55:17 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 08 09:57:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.20.18@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 09:57:30 fir-md1-s1 kernel: LustreError: Skipped 16 previous similar messages Jul 08 10:00:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 10:00:09 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 08 10:03:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 10:03:35 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 08 10:04:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 10:04:03 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 08 10:10:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 10:10:26 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 08 10:10:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 10:10:51 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 08 10:14:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 10:14:58 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 08 10:15:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 10:15:07 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 08 10:20:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 10:20:34 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 08 10:24:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 10:24:22 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 08 10:25:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 10:25:16 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 08 10:25:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 10:25:18 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 08 10:30:38 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 10:30:38 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 08 10:35:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 10:35:01 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 08 10:35:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 10:35:20 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 08 10:35:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 10:35:27 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 08 10:40:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 10:40:45 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 08 10:45:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 10:45:31 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 08 10:45:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 10:45:52 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 08 10:51:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 10:51:01 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 08 10:56:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 10:56:08 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 08 10:57:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.17.11@o2ib6, removing former export from same NID Jul 08 10:57:12 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 08 10:59:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 10:59:42 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 08 11:01:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 08 11:01:04 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 08 11:01:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 11:01:46 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 11:04:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client a2700990-6487-6425-0ded-6ef948a9753e (at 10.8.30.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f8b3e000, cur 1562609096 expire 1562608946 last 1562608869 Jul 08 11:04:56 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 11:06:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 11:06:28 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 08 11:06:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 11:06:49 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 08 11:07:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 11:07:17 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 08 11:11:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 11:11:15 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 08 11:14:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 11:14:45 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 11:16:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 11:16:49 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 08 11:17:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 11:17:20 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 08 11:21:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 11:21:20 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 08 11:27:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 11:27:19 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 08 11:27:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 11:27:26 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 08 11:31:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 11:31:06 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 08 11:31:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 11:31:24 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 08 11:37:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 11:37:26 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 08 11:37:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 11:37:39 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 08 11:42:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 11:42:37 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 08 11:47:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 11:47:41 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 08 11:48:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 11:48:01 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 08 11:48:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 11:48:26 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 11:52:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 11:52:48 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 11:58:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 11:58:52 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 08 11:58:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 11:58:54 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 08 12:03:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 12:03:05 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 08 12:08:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 12:08:54 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 12:10:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 12:10:31 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 08 12:11:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 12:11:54 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 12:13:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 12:13:15 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 08 12:19:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 12:19:05 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 08 12:20:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 12:20:25 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 12:20:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 12:20:35 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 08 12:23:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 12:23:31 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 08 12:29:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 12:29:18 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 08 12:30:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 12:30:44 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 08 12:33:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 12:33:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 12:33:42 fir-md1-s1 kernel: Lustre: Skipped 104 previous similar messages Jul 08 12:36:46 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2534fc0400, cur 1562614606 expire 1562614456 last 1562614379 Jul 08 12:36:46 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 08 12:37:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 12:37:23 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 12:38:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 12:39:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 12:39:56 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 08 12:40:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 12:40:10 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 12:40:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 12:40:47 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 08 12:44:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 12:44:07 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 12:44:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 12:44:27 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 08 12:48:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148ce91c00, cur 1562615302 expire 1562615152 last 1562615075 Jul 08 12:50:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 12:50:15 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 08 12:50:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 12:50:50 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 08 12:53:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1fd98ce000, cur 1562615597 expire 1562615447 last 1562615370 Jul 08 12:54:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 12:54:37 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 08 12:56:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 12:56:39 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 08 13:00:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 13:00:29 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 08 13:01:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 13:01:09 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 08 13:04:39 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 13:04:39 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jul 08 13:11:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 13:11:12 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 08 13:11:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 13:11:18 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 08 13:13:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 13:13:47 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 08 13:14:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 08 13:14:45 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 08 13:21:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 13:21:21 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 08 13:21:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 13:21:25 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 13:24:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 13:24:01 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 08 13:24:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 13:24:55 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 08 13:25:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client e488b13f-c3ed-66de-0053-32b5151ace52 (at 10.8.15.6@o2ib6) in 192 seconds. I think it's dead, and I am evicting it. exp ffff8f229dea1000, cur 1562617549 expire 1562617399 last 1562617357 Jul 08 13:26:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e488b13f-c3ed-66de-0053-32b5151ace52 (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f229dea2c00, cur 1562617584 expire 1562617434 last 1562617357 Jul 08 13:31:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 13:31:33 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 08 13:33:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 08 13:33:26 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 08 13:34:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 13:34:44 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 08 13:35:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 13:35:00 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 08 13:41:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 13:41:36 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 13:45:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 13:45:03 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 08 13:45:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 13:45:03 fir-md1-s1 kernel: Lustre: Skipped 104 previous similar messages Jul 08 13:50:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 13:50:30 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 08 13:51:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 13:51:55 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 08 13:55:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 13:55:28 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 08 13:55:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 13:55:53 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 08 14:01:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 14:01:15 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 08 14:02:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 14:02:39 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 08 14:05:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 08 14:05:32 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 08 14:06:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 14:06:00 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 14:12:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 14:12:43 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 08 14:14:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 14:14:56 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 08 14:15:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 14:15:49 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 08 14:18:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 14:18:06 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 08 14:22:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 14:22:55 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 08 14:25:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 14:25:46 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 08 14:26:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 14:26:05 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 08 14:29:19 fir-md1-s1 kernel: Lustre: 23730:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562621348/real 1562621348] req@ffff8f37212cd400 x1636727036156160/t0(0) o104->fir-MDT0000@10.8.11.6@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562621359 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 14:29:19 fir-md1-s1 kernel: Lustre: 23730:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 08 14:29:23 fir-md1-s1 kernel: Lustre: 21378:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3bcaa46300 x1633654418410912/t0(0) o36->60a9f157-4802-e53d-dccf-19f0d690f2d1@10.9.0.1@o2ib4:28/0 lens 496/448 e 1 to 0 dl 1562621368 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 14:29:23 fir-md1-s1 kernel: Lustre: 21378:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 574 previous similar messages Jul 08 14:29:24 fir-md1-s1 kernel: Lustre: 21378:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f428476fb00 x1631569431335360/t0(0) o101->20b94f29-3d6d-5fdd-bf3c-536686b5a4fe@10.9.107.47@o2ib4:29/0 lens 576/0 e 1 to 0 dl 1562621369 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 08 14:29:24 fir-md1-s1 kernel: Lustre: 21378:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 252 previous similar messages Jul 08 14:29:25 fir-md1-s1 kernel: Lustre: 21378:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f201b773600 x1631609088794304/t0(0) o101->c816839b-680c-9a56-ca6b-6b0e082ba795@10.9.106.34@o2ib4:0/0 lens 576/0 e 1 to 0 dl 1562621370 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 08 14:29:25 fir-md1-s1 kernel: Lustre: 21378:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 107 previous similar messages Jul 08 14:29:27 fir-md1-s1 kernel: Lustre: 23570:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0b9815d700 x1634169439253792/t0(0) o101->97b378ef-cbc6-b9bf-0007-7fdb21d6a3a7@10.9.109.23@o2ib4:2/0 lens 576/0 e 1 to 0 dl 1562621372 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 08 14:29:27 fir-md1-s1 kernel: Lustre: 23570:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 156 previous similar messages Jul 08 14:29:30 fir-md1-s1 kernel: Lustre: 23727:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:2s); client may timeout. req@ffff8f3bcaa40600 x1634079367746016/t0(0) o101->49aa8323-a38d-3237-508c-ea94c68aa863@10.9.108.53@o2ib4:28/0 lens 576/0 e 1 to 0 dl 1562621368 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 08 14:29:30 fir-md1-s1 kernel: LustreError: 21128:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.101.27@o2ib4: deadline 20:1s ago req@ffff8f43fdbea700 x1631659625120224/t0(0) o101->b7aae4ae-1aa0-9e5d-5ecf-90e4dbcd33de@10.9.101.27@o2ib4:29/0 lens 576/0 e 1 to 0 dl 1562621369 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 08 14:29:30 fir-md1-s1 kernel: LustreError: 21128:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jul 08 14:29:30 fir-md1-s1 kernel: Lustre: 23727:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 267 previous similar messages Jul 08 14:30:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 14:30:29 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 08 14:33:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 14:33:36 fir-md1-s1 kernel: Lustre: Skipped 727 previous similar messages Jul 08 14:36:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 14:36:18 fir-md1-s1 kernel: Lustre: Skipped 759 previous similar messages Jul 08 14:40:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 14:40:16 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 14:41:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 14:41:14 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 08 14:43:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 14:43:48 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 08 14:46:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 08 14:46:32 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 08 14:51:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 14:51:22 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 08 14:53:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 14:53:43 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 14:54:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 14:54:07 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 08 14:57:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 14:57:10 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 08 15:02:18 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3385afe000, cur 1562623338 expire 1562623188 last 1562623111 Jul 08 15:02:18 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 08 15:03:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 15:03:44 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 08 15:04:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 15:04:36 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 08 15:04:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 15:04:39 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 08 15:08:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 08 15:08:00 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 08 15:15:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 15:15:17 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 08 15:16:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 15:16:01 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 15:18:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 15:18:22 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 08 15:19:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 15:19:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 15:19:01 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 08 15:26:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 15:26:27 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 15:26:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 15:26:30 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 08 15:28:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 15:28:26 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 08 15:29:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 15:29:47 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 08 15:32:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25211cbc00, cur 1562625132 expire 1562624982 last 1562624905 Jul 08 15:37:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 15:37:01 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 08 15:37:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 15:37:18 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 08 15:38:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 15:38:27 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 08 15:47:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 15:47:14 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 15:47:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 15:47:23 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 08 15:48:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 15:48:34 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 08 15:51:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 15:52:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 15:52:41 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 15:58:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 15:58:02 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 08 15:58:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 15:58:45 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 08 15:59:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 15:59:46 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 08 16:06:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 16:06:23 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 16:08:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 16:08:28 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 16:08:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 16:08:55 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 08 16:10:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 16:10:16 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 08 16:18:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 16:18:40 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 08 16:19:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 16:19:06 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 08 16:21:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 16:21:06 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 08 16:28:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 16:28:50 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 08 16:29:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 16:29:17 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 08 16:32:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 16:32:29 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 08 16:36:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 16:36:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 8447a07a-e92a-94fe-737c-da4e88830639 (at 10.9.107.43@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1a5e53a800, cur 1562628969 expire 1562628819 last 1562628742 Jul 08 16:37:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 16:38:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 16:39:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 16:39:01 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 08 16:40:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 16:40:23 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 08 16:40:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 4dda764c-5ca7-3340-a1d3-17b756c64805 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1644fc6800, cur 1562629259 expire 1562629109 last 1562629032 Jul 08 16:40:59 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 16:42:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 16:42:45 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 16:49:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 16:49:43 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 08 16:50:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 16:50:36 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 08 16:53:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 16:53:34 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 08 16:54:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 16:55:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 16:56:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:00:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 17:00:03 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 08 17:01:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 17:01:06 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 08 17:04:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 17:04:00 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 08 17:10:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 17:10:25 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 08 17:11:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 17:11:13 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 08 17:14:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 17:14:05 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 08 17:20:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 17:20:38 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 08 17:21:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 17:21:22 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 08 17:24:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 17:24:35 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 08 17:28:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:31:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 17:31:04 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 08 17:31:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 17:31:22 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 08 17:34:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 17:34:36 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 08 17:35:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:36:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:36:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:37:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:40:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:41:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 17:41:14 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 08 17:41:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 17:41:30 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 08 17:43:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:44:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:44:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 17:44:49 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 08 17:46:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:46:40 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 08 17:49:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:49:12 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 08 17:51:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 17:51:48 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 08 17:51:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 17:51:48 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 08 17:54:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 17:54:26 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 08 17:55:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 17:55:53 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 08 18:02:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 18:02:10 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 08 18:02:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 18:02:42 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 08 18:04:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 18:04:43 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 08 18:08:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 18:08:07 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 18:12:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 18:12:12 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 08 18:12:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 18:12:58 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 18:15:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 18:15:25 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 08 18:18:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 18:18:22 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 08 18:22:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 08 18:22:24 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 08 18:23:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 18:23:17 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 08 18:26:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 18:26:03 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 08 18:30:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 18:30:13 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 08 18:32:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 18:32:28 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 18:33:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 18:33:38 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 08 18:36:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 18:36:55 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 08 18:41:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 18:41:48 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 08 18:42:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 18:42:28 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 08 18:43:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 18:43:41 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 08 18:46:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 18:46:58 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 08 18:51:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 08 18:51:53 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 08 18:52:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 18:52:34 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 08 18:53:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 18:53:57 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 18:57:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 18:57:11 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 08 19:02:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 19:02:34 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 08 19:02:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 19:02:44 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 08 19:04:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 19:04:42 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 19:06:15 fir-md1-s1 kernel: Lustre: 22280:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1bef6cd700 x1631840018373024/t0(0) o101->533f2d59-21df-dd34-d3a6-f780aca8b580@10.8.25.3@o2ib6:20/0 lens 480/568 e 0 to 0 dl 1562637980 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 19:06:15 fir-md1-s1 kernel: Lustre: 22280:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 178 previous similar messages Jul 08 19:06:19 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2e007c1f80/0x5d9ee63602816224 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 13 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55c8bda65 expref: 43 pid: 10143 timeout: 1753039 lvb_type: 0 Jul 08 19:07:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 19:07:50 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 08 19:10:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2af4e29400, cur 1562638215 expire 1562638065 last 1562637988 Jul 08 19:10:15 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 19:12:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 19:12:44 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 08 19:14:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 19:14:05 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 08 19:14:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 19:14:53 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 08 19:18:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 19:18:14 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 08 19:22:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 19:22:47 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 08 19:24:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 19:24:11 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 08 19:25:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 19:25:21 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 19:30:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 19:30:18 fir-md1-s1 kernel: LustreError: Skipped 14 previous similar messages Jul 08 19:33:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 19:33:06 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 08 19:34:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 19:34:41 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 08 19:35:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 19:35:50 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 19:40:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 19:40:50 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 08 19:43:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 19:43:09 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 08 19:45:52 fir-md1-s1 kernel: Lustre: 23710:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f3e497b3c00 x1631318301775872/t0(0) o101->ddef0525-fd05-baf0-eec8-55af7a82431b@10.8.24.4@o2ib6:27/0 lens 480/568 e 0 to 0 dl 1562640357 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 19:45:57 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.30.19@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1e0776d580/0x5d9ee636114fc7ba lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 16 type: IBT flags: 0x60200400000020 nid: 10.8.30.19@o2ib6 remote: 0xcd8d918f46f6186b expref: 46 pid: 97644 timeout: 1755417 lvb_type: 0 Jul 08 19:46:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 19:46:26 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 08 19:46:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 08 19:46:42 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 08 19:52:17 fir-md1-s1 kernel: Lustre: 22280:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1e9e830c00 x1631318302836336/t0(0) o101->ddef0525-fd05-baf0-eec8-55af7a82431b@10.8.24.4@o2ib6:22/0 lens 480/568 e 1 to 0 dl 1562640742 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 19:52:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 19:52:21 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 08 19:52:28 fir-md1-s1 kernel: Lustre: 23743:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2885688300 x1631827979371248/t0(0) o101->d3f5a92e-e73a-b021-4354-c2176911d60c@10.8.30.19@o2ib6:3/0 lens 480/568 e 0 to 0 dl 1562640753 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 19:52:56 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f30d5640f00 x1633783498073088/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:1/0 lens 480/568 e 0 to 0 dl 1562640781 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 19:52:56 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 19:53:00 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2753adec00/0x5d9ee63613f09f8d lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x75:0x0].0x0 bits 0x40/0x0 rrc: 22 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55c98ca61 expref: 50 pid: 21679 timeout: 1755840 lvb_type: 0 Jul 08 19:53:10 fir-md1-s1 kernel: Lustre: 23704:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2fcbd5bc00 x1631827979378736/t0(0) o101->d3f5a92e-e73a-b021-4354-c2176911d60c@10.8.30.19@o2ib6:15/0 lens 480/568 e 0 to 0 dl 1562640795 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 19:53:10 fir-md1-s1 kernel: Lustre: 23704:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 19:53:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b4e8a7c4-09eb-baae-5220-9b1baa9441aa (at 10.8.30.19@o2ib6) Jul 08 19:53:16 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 08 19:53:30 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.24.4@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f324c2f8d80/0x5d9ee63613f18bdb lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x75:0x0].0x0 bits 0x40/0x0 rrc: 17 type: IBT flags: 0x60200400000020 nid: 10.8.24.4@o2ib6 remote: 0x8a5ac3af8bf42be6 expref: 40 pid: 23704 timeout: 1755870 lvb_type: 0 Jul 08 19:54:00 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.25.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2cc9bc9d40/0x5d9ee63613f245a1 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x75:0x0].0x0 bits 0x40/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd263afcb expref: 847 pid: 23748 timeout: 1755900 lvb_type: 0 Jul 08 19:54:01 fir-md1-s1 kernel: LustreError: 97644:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1fceb30000 x1636727138629184/t0(0) o104->fir-MDT0002@10.8.25.23@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 08 19:54:01 fir-md1-s1 kernel: LustreError: 97654:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f451bb7cc00 ns: mdt-fir-MDT0002_UUID lock: ffff8f1c9ba2f740/0x5d9ee63614703e9a lrc: 3/0,0 mode: PW/PW res: [0x2c002c126:0x3e:0x0].0x0 bits 0x40/0x0 rrc: 14 type: IBT flags: 0x50200000000000 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd26dd78e expref: 384 pid: 97654 timeout: 0 lvb_type: 0 Jul 08 19:56:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 19:56:37 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 08 19:57:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 19:57:33 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 08 20:03:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 08 20:03:38 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 08 20:04:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 20:04:30 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 08 20:06:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 20:06:50 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 08 20:07:05 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562641618/real 1562641618] req@ffff8f28367f0f00 x1636727143978848/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562641625 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 20:07:05 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 08 20:07:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 20:07:44 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 08 20:10:05 fir-md1-s1 kernel: Lustre: 23733:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562641798/real 1562641798] req@ffff8f3468ae5400 x1636727145014864/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562641805 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 20:13:40 fir-md1-s1 kernel: Lustre: 21420:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1fe3246c00 x1634176436499536/t0(0) o101->bff671a6-6393-a53b-8c2a-0f521cd0a513@10.9.109.13@o2ib4:15/0 lens 1768/3288 e 1 to 0 dl 1562642025 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 20:13:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 08 20:13:56 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 08 20:14:11 fir-md1-s1 kernel: Lustre: 20511:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f22d137c800 x1631318304603760/t0(0) o101->ddef0525-fd05-baf0-eec8-55af7a82431b@10.8.24.4@o2ib6:16/0 lens 480/568 e 1 to 0 dl 1562642056 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 20:14:20 fir-md1-s1 kernel: Lustre: 24581:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f24d5a89800 x1633783517478704/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:25/0 lens 480/568 e 0 to 0 dl 1562642065 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 20:14:20 fir-md1-s1 kernel: Lustre: 24581:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 20:14:25 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f23d98cf500/0x5d9ee6361c267447 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 21 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55c9bcc61 expref: 19 pid: 50446 timeout: 1757125 lvb_type: 0 Jul 08 20:14:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 20:14:43 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 08 20:14:55 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.25.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f230e1dee40/0x5d9ee6361c26f68b lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 16 type: IBT flags: 0x60200400000020 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd2bfc1c7 expref: 217 pid: 26256 timeout: 1757155 lvb_type: 0 Jul 08 20:14:55 fir-md1-s1 kernel: LustreError: 21679:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f30cf2bd400 ns: mdt-fir-MDT0002_UUID lock: ffff8f2873f63600/0x5d9ee6361c8a35f2 lrc: 3/0,0 mode: PW/PW res: [0x2c002c406:0x3:0x0].0x0 bits 0x40/0x0 rrc: 10 type: IBT flags: 0x50200000000000 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd2c58c77 expref: 135 pid: 21679 timeout: 0 lvb_type: 0 Jul 08 20:14:55 fir-md1-s1 kernel: LustreError: 31015:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.25.23@o2ib6 arrived at 1562642095 with bad export cookie 6746082411793135065 Jul 08 20:14:55 fir-md1-s1 kernel: LustreError: 21679:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message Jul 08 20:15:25 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.24.4@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f204f088240/0x5d9ee6361c27850f lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.8.24.4@o2ib6 remote: 0x8a5ac3af8bfbb244 expref: 19 pid: 22287 timeout: 1757185 lvb_type: 0 Jul 08 20:16:24 fir-md1-s1 kernel: Lustre: 24581:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1e0841b000 x1631318304621008/t0(0) o101->ddef0525-fd05-baf0-eec8-55af7a82431b@10.8.24.4@o2ib6:29/0 lens 480/568 e 1 to 0 dl 1562642189 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 20:16:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client ddef0525-fd05-baf0-eec8-55af7a82431b (at 10.8.24.4@o2ib6) reconnecting Jul 08 20:16:51 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 08 20:17:34 fir-md1-s1 kernel: LustreError: 23739:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562642164, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f330a250240/0x5d9ee6361d05c501 lrc: 3/0,1 mode: --/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 19 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 23739 timeout: 0 lvb_type: 0 Jul 08 20:17:34 fir-md1-s1 kernel: LustreError: 23739:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 08 20:17:35 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.19@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f214755a1c0/0x5d9ee6361d028da8 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 19 type: IBT flags: 0x60200400000020 nid: 10.8.30.19@o2ib6 remote: 0xcd8d918f4702d808 expref: 19 pid: 24583 timeout: 1757315 lvb_type: 0 Jul 08 20:17:39 fir-md1-s1 kernel: LustreError: 97672:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562642169, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f221fa1fbc0/0x5d9ee6361d0cf34c lrc: 3/0,1 mode: --/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 14 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97672 timeout: 0 lvb_type: 0 Jul 08 20:17:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 20:17:53 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 08 20:19:20 fir-md1-s1 kernel: Lustre: 23704:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f28862d6c00 x1631827981350560/t0(0) o101->d3f5a92e-e73a-b021-4354-c2176911d60c@10.8.30.19@o2ib6:25/0 lens 480/568 e 0 to 0 dl 1562642365 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 20:19:20 fir-md1-s1 kernel: Lustre: 23704:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 08 20:19:24 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1e50a86c00/0x5d9ee6361e1c8f72 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x75:0x0].0x0 bits 0x40/0x0 rrc: 21 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55c9bddf6 expref: 19 pid: 97672 timeout: 1757424 lvb_type: 0 Jul 08 20:19:54 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.30.19@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f0527a072c0/0x5d9ee6361e1d158a lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x75:0x0].0x0 bits 0x40/0x0 rrc: 16 type: IBT flags: 0x60200400000020 nid: 10.8.30.19@o2ib6 remote: 0xcd8d918f47038bae expref: 19 pid: 23692 timeout: 1757454 lvb_type: 0 Jul 08 20:23:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 20:23:57 fir-md1-s1 kernel: Lustre: Skipped 115 previous similar messages Jul 08 20:24:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 20:24:59 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 08 20:26:13 fir-md1-s1 kernel: Lustre: 23733:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562642766/real 1562642766] req@ffff8f2fc5d36c00 x1636727151419936/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562642773 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 20:26:40 fir-md1-s1 kernel: Lustre: 97664:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2129fc2d00 x1631827982161376/t0(0) o101->d3f5a92e-e73a-b021-4354-c2176911d60c@10.8.30.19@o2ib6:15/0 lens 480/568 e 0 to 0 dl 1562642805 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 20:26:40 fir-md1-s1 kernel: Lustre: 97664:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 08 20:26:44 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.25.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1a1686b3c0/0x5d9ee636209c8467 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 19 type: IBT flags: 0x60200400000020 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd307bed6 expref: 194 pid: 23733 timeout: 1757864 lvb_type: 0 Jul 08 20:26:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 20:26:55 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 08 20:28:19 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3489786800, cur 1562642899 expire 1562642749 last 1562642672 Jul 08 20:31:34 fir-md1-s1 kernel: Lustre: 23652:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f34ccddad00 x1631318306240656/t0(0) o101->ddef0525-fd05-baf0-eec8-55af7a82431b@10.8.24.4@o2ib6:9/0 lens 480/568 e 0 to 0 dl 1562643099 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 20:31:34 fir-md1-s1 kernel: Lustre: 23652:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 20:31:38 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.19@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f348c740b40/0x5d9ee636224a4ea6 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x75:0x0].0x0 bits 0x40/0x0 rrc: 19 type: IBT flags: 0x60200400000020 nid: 10.8.30.19@o2ib6 remote: 0xcd8d918f47099e48 expref: 20 pid: 21380 timeout: 1758158 lvb_type: 0 Jul 08 20:32:24 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562643137/real 1562643137] req@ffff8f2af9ef3600 x1636727153982944/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562643144 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 20:33:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 20:33:14 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 08 20:35:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 20:35:03 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 08 20:35:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 20:35:08 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 08 20:37:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 20:37:16 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 08 20:43:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 20:43:32 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 08 20:45:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 20:45:18 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 08 20:45:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 20:45:19 fir-md1-s1 kernel: LustreError: Skipped 20 previous similar messages Jul 08 20:47:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 20:47:30 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 08 20:52:53 fir-md1-s1 kernel: Lustre: 23687:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562644366/real 1562644366] req@ffff8f0d3e72a100 x1636727162900240/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562644373 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 20:53:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 20:53:43 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 08 20:55:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 08 20:55:29 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 08 20:56:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 20:56:02 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 08 20:57:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 20:57:43 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 08 21:00:12 fir-md1-s1 kernel: Lustre: 21003:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f279a4f1500 x1631596157296432/t0(0) o101->169021a4-a808-827d-1880-f3d0a2ab5ac3@10.9.103.20@o2ib4:17/0 lens 480/568 e 1 to 0 dl 1562644817 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:00:12 fir-md1-s1 kernel: Lustre: 21003:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 21:00:26 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1e94a74140/0x5d9ee6362cc520ea lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x75:0x0].0x0 bits 0x40/0x0 rrc: 22 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55c9e43c6 expref: 19 pid: 97654 timeout: 1759886 lvb_type: 0 Jul 08 21:00:44 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 08 21:00:45 fir-md1-s1 kernel: Lustre: 23689:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 08 21:00:45 fir-md1-s1 kernel: Lustre: 23689:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 261 previous similar messages Jul 08 21:00:46 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 08 21:00:46 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 243 previous similar messages Jul 08 21:00:49 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 08 21:00:49 fir-md1-s1 kernel: Lustre: 21420:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 469 previous similar messages Jul 08 21:00:56 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.9.103.20@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f2fb158d7c0/0x5d9ee6362cc553fe lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x75:0x0].0x0 bits 0x40/0x0 rrc: 17 type: IBT flags: 0x60200400000020 nid: 10.9.103.20@o2ib4 remote: 0x8e6a8fb7733dfaab expref: 277 pid: 23608 timeout: 1759916 lvb_type: 0 Jul 08 21:03:26 fir-md1-s1 kernel: Lustre: 10198:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0a0d910f00 x1631596157941024/t0(0) o101->169021a4-a808-827d-1880-f3d0a2ab5ac3@10.9.103.20@o2ib4:1/0 lens 480/568 e 1 to 0 dl 1562645011 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:03:26 fir-md1-s1 kernel: Lustre: 10198:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 21:03:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 21:03:47 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 08 21:04:07 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.9.103.20@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f06fc992880/0x5d9ee6362de4771a lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 20 type: IBT flags: 0x60200400000020 nid: 10.9.103.20@o2ib4 remote: 0x8e6a8fb773412195 expref: 34 pid: 23618 timeout: 1760107 lvb_type: 0 Jul 08 21:04:08 fir-md1-s1 kernel: LustreError: 23103:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.9.103.20@o2ib4 arrived at 1562645048 with bad export cookie 6746082412206550431 Jul 08 21:05:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 08 21:05:46 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 08 21:06:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 21:06:55 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 08 21:08:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 21:08:00 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 08 21:10:16 fir-md1-s1 kernel: Lustre: 21680:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562645409/real 1562645409] req@ffff8f36e667c800 x1636727169387760/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562645416 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 21:12:35 fir-md1-s1 kernel: Lustre: 22288:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562645548/real 1562645548] req@ffff8f2215a59e00 x1636727170095040/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562645555 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 21:13:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 21:13:48 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 08 21:16:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 21:16:12 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 08 21:17:37 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3e516bf000, cur 1562645857 expire 1562645707 last 1562645630 Jul 08 21:18:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 21:18:06 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 08 21:18:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 21:18:26 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 08 21:19:23 fir-md1-s1 kernel: Lustre: 22282:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f250c4e2100 x1631596161683808/t0(0) o101->169021a4-a808-827d-1880-f3d0a2ab5ac3@10.9.103.20@o2ib4:28/0 lens 480/568 e 0 to 0 dl 1562645968 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:19:23 fir-md1-s1 kernel: Lustre: 22282:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 21:19:27 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f34a3a45e80/0x5d9ee63633f88b91 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 22 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55ca01133 expref: 19 pid: 23664 timeout: 1761027 lvb_type: 0 Jul 08 21:19:57 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.25.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f19f0a4bcc0/0x5d9ee63633f9e195 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 17 type: IBT flags: 0x60200400000020 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd4173abf expref: 169 pid: 22288 timeout: 1761057 lvb_type: 0 Jul 08 21:23:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 21:23:56 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 08 21:26:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 21:26:13 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 08 21:28:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 21:28:44 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 08 21:28:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 21:28:58 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 08 21:32:17 fir-md1-s1 kernel: Lustre: 21368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562646729/real 1562646729] req@ffff8f08e6547500 x1636727177556416/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562646736 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 21:32:23 fir-md1-s1 kernel: Lustre: 10143:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562646736/real 1562646736] req@ffff8f348c681200 x1636727177577648/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562646743 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 21:34:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 21:34:41 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 08 21:35:41 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562646934/real 1562646934] req@ffff8f344d2ca400 x1636727178440112/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562646941 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 21:36:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 08 21:36:17 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 08 21:38:06 fir-md1-s1 kernel: Lustre: 21413:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f349849fb00 x1633783605451072/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:11/0 lens 480/568 e 0 to 0 dl 1562647091 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:38:06 fir-md1-s1 kernel: Lustre: 21413:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 21:38:10 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2e6ff50900/0x5d9ee6363b5914a1 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55ca325fc expref: 57 pid: 23710 timeout: 1762150 lvb_type: 0 Jul 08 21:38:44 fir-md1-s1 kernel: Lustre: 97652:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1d8f7bfb00 x1633783606210032/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:19/0 lens 480/568 e 0 to 0 dl 1562647129 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:38:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 21:38:47 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 08 21:39:39 fir-md1-s1 kernel: Lustre: 23601:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f26008ddd00 x1633783606826736/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:14/0 lens 480/568 e 0 to 0 dl 1562647184 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:39:43 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1caecc2640/0x5d9ee6363c0b947e lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55ca33426 expref: 19 pid: 24585 timeout: 1762243 lvb_type: 0 Jul 08 21:40:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 21:40:11 fir-md1-s1 kernel: LustreError: Skipped 13 previous similar messages Jul 08 21:42:36 fir-md1-s1 kernel: Lustre: 21460:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1cefb3ef00 x1633783610784720/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:11/0 lens 480/568 e 0 to 0 dl 1562647361 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:45:46 fir-md1-s1 kernel: Lustre: 20730:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1ce67e0900 x1633783615047440/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:21/0 lens 480/568 e 0 to 0 dl 1562647551 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:45:50 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f34622d72c0/0x5d9ee6363e7a0f5b lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55ca33f1d expref: 19 pid: 23748 timeout: 1762610 lvb_type: 0 Jul 08 21:46:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 21:46:40 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 08 21:47:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 21:47:44 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 08 21:48:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 21:48:53 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 08 21:50:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 21:50:30 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 08 21:53:38 fir-md1-s1 kernel: Lustre: 23687:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f09b6b27500 x1631538537486240/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:13/0 lens 480/568 e 0 to 0 dl 1562648023 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 21:53:42 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f16b6d8a640/0x5d9ee636419d63c1 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 16 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55ca4b358 expref: 19 pid: 21413 timeout: 1763082 lvb_type: 0 Jul 08 21:54:12 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.9.103.34@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f13a9344140/0x5d9ee636419da4e3 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.9.103.34@o2ib4 remote: 0x479fd480650c933e expref: 151 pid: 23612 timeout: 1763112 lvb_type: 0 Jul 08 21:54:13 fir-md1-s1 kernel: LustreError: 23554:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f1488901000 ns: mdt-fir-MDT0002_UUID lock: ffff8f10de4933c0/0x5d9ee6364200bbf5 lrc: 3/0,0 mode: PW/PW res: [0x2c002c148:0x7d:0x0].0x0 bits 0x40/0x0 rrc: 11 type: IBT flags: 0x50200000000000 nid: 10.9.103.34@o2ib4 remote: 0x479fd480650d9db5 expref: 27 pid: 23554 timeout: 0 lvb_type: 0 Jul 08 21:54:13 fir-md1-s1 kernel: LustreError: 23554:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages Jul 08 21:56:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 08 21:56:46 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 08 21:57:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 21:57:46 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 08 21:58:24 fir-md1-s1 kernel: Lustre: 20511:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562648297/real 1562648297] req@ffff8f2215a5aa00 x1636727187978544/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562648304 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 21:59:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 21:59:03 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 08 22:01:49 fir-md1-s1 kernel: Lustre: 10143:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562648502/real 1562648502] req@ffff8f34a92ca700 x1636727189714544/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562648509 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 22:02:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 22:02:28 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 08 22:03:11 fir-md1-s1 kernel: Lustre: 27320:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562648584/real 1562648584] req@ffff8f0b5a8c8000 x1636727190085616/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562648591 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 22:04:23 fir-md1-s1 kernel: Lustre: 23711:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562648656/real 1562648656] req@ffff8f3e28a00300 x1636727190501888/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562648663 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 22:06:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 08 22:06:48 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 08 22:08:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 08 22:08:06 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 08 22:09:19 fir-md1-s1 kernel: Lustre: 23618:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0eb9dcce00 x1631538539899360/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:24/0 lens 480/568 e 1 to 0 dl 1562648964 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:09:19 fir-md1-s1 kernel: Lustre: 23618:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 22:09:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8 (at 10.9.103.34@o2ib4) reconnecting Jul 08 22:09:25 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 08 22:09:29 fir-md1-s1 kernel: Lustre: 23645:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2fce759500 x1633783633323312/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:4/0 lens 480/568 e 0 to 0 dl 1562648974 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:09:33 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1eceb01f80/0x5d9ee6364793a473 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 16 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55ca7c0a3 expref: 19 pid: 24585 timeout: 1764033 lvb_type: 0 Jul 08 22:12:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 22:12:43 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 08 22:16:05 fir-md1-s1 kernel: Lustre: 23618:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f0fd4606900 x1631538540468448/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:10/0 lens 480/568 e 0 to 0 dl 1562649370 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:16:09 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.25.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f3321d0cec0/0x5d9ee6364a32b6dc lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 14 type: IBT flags: 0x60200400000020 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd4e438a7 expref: 391 pid: 23608 timeout: 1764429 lvb_type: 0 Jul 08 22:16:09 fir-md1-s1 kernel: LustreError: 25079:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.25.23@o2ib6 arrived at 1562649369 with bad export cookie 6746082412328613743 Jul 08 22:16:10 fir-md1-s1 kernel: LustreError: 23723:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f237ea3e800 ns: mdt-fir-MDT0002_UUID lock: ffff8f287e35b840/0x5d9ee6364a6868c1 lrc: 3/0,0 mode: PW/PW res: [0x2c002c299:0x76:0x0].0x0 bits 0x40/0x0 rrc: 12 type: IBT flags: 0x50200000000000 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd4e7f761 expref: 10 pid: 23723 timeout: 0 lvb_type: 0 Jul 08 22:17:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 22:17:05 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 08 22:18:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 08 22:18:53 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 08 22:19:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 22:19:54 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 08 22:24:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 22:24:03 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 08 22:24:24 fir-md1-s1 kernel: Lustre: 23579:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f36d1a3b300 x1631538542400400/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:29/0 lens 480/568 e 0 to 0 dl 1562649869 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:24:28 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f23088eba80/0x5d9ee6364d70cfbd lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 21 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55ca9f92b expref: 19 pid: 97654 timeout: 1764928 lvb_type: 0 Jul 08 22:25:16 fir-md1-s1 kernel: Lustre: 23748:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f26a6721e00 x1631776452617600/t0(0) o101->cb1e051f-12ef-c393-c1de-bc60ba01debc@10.8.13.11@o2ib6:21/0 lens 480/568 e 0 to 0 dl 1562649921 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:25:16 fir-md1-s1 kernel: Lustre: 23748:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 08 22:25:20 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.25.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1f4f487bc0/0x5d9ee6364db317bd lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 17 type: IBT flags: 0x60200400000020 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd51b49dc expref: 106 pid: 20730 timeout: 1764980 lvb_type: 0 Jul 08 22:25:20 fir-md1-s1 kernel: LustreError: 25030:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.25.23@o2ib6 arrived at 1562649920 with bad export cookie 6746082412698566123 Jul 08 22:25:20 fir-md1-s1 kernel: LustreError: 21003:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f1a2f6c6000 ns: mdt-fir-MDT0002_UUID lock: ffff8f263bf23a80/0x5d9ee6364de32c02 lrc: 3/0,0 mode: PW/PW res: [0x2c002c408:0x4:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x50200000000000 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd51ee12f expref: 24 pid: 21003 timeout: 0 lvb_type: 0 Jul 08 22:25:20 fir-md1-s1 kernel: LustreError: 21003:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message Jul 08 22:25:50 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.9.103.34@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f122353f980/0x5d9ee6364db42cbb lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.9.103.34@o2ib4 remote: 0x479fd480651f07ce expref: 43 pid: 22280 timeout: 1765010 lvb_type: 0 Jul 08 22:25:51 fir-md1-s1 kernel: LustreError: 23632:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f2fd1ddc400 ns: mdt-fir-MDT0002_UUID lock: ffff8f07d5ff2d00/0x5d9ee6364e158666 lrc: 3/0,0 mode: PW/PW res: [0x2c002c148:0x7e:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x50200000000000 nid: 10.9.103.34@o2ib4 remote: 0x479fd480651fd36f expref: 8 pid: 23632 timeout: 0 lvb_type: 0 Jul 08 22:25:51 fir-md1-s1 kernel: LustreError: 23632:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message Jul 08 22:27:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 22:27:33 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 08 22:28:43 fir-md1-s1 kernel: Lustre: 23686:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f35e0a4e900 x1631538543215680/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:18/0 lens 480/568 e 1 to 0 dl 1562650128 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:28:43 fir-md1-s1 kernel: Lustre: 23686:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 22:30:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 22:30:00 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 08 22:30:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 22:30:36 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 08 22:30:39 fir-md1-s1 kernel: Lustre: 97665:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562650232/real 1562650232] req@ffff8f1f6a956c00 x1636727203823808/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562650239 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 22:32:42 fir-md1-s1 kernel: Lustre: 23571:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f0e3be98300 x1631538543770800/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:17/0 lens 480/568 e 0 to 0 dl 1562650367 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:32:42 fir-md1-s1 kernel: Lustre: 23571:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 08 22:32:46 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.25.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2122373a80/0x5d9ee636507a323b lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 18 type: IBT flags: 0x60200400000020 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd541da60 expref: 146 pid: 50446 timeout: 1765426 lvb_type: 0 Jul 08 22:32:47 fir-md1-s1 kernel: LustreError: 23678:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f1e5214b400 ns: mdt-fir-MDT0002_UUID lock: ffff8f2604608fc0/0x5d9ee63650af3e73 lrc: 3/0,0 mode: PW/PW res: [0x2c002c11e:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x50200000000000 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd545183b expref: 63 pid: 23678 timeout: 0 lvb_type: 0 Jul 08 22:32:47 fir-md1-s1 kernel: LustreError: 23678:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 1 previous similar message Jul 08 22:34:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 22:34:04 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 08 22:35:40 fir-md1-s1 kernel: Lustre: 27320:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562650533/real 1562650533] req@ffff8f4328321b00 x1636727205751232/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562650540 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 22:37:21 fir-md1-s1 kernel: Lustre: 23645:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562650634/real 1562650634] req@ffff8f34415e6c00 x1636727206274400/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562650641 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 22:38:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 22:38:14 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 08 22:40:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 22:40:07 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 08 22:40:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 22:40:42 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 08 22:41:26 fir-md1-s1 kernel: Lustre: 23678:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2898bbcb00 x1633783665580720/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:1/0 lens 480/568 e 1 to 0 dl 1562650891 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:41:26 fir-md1-s1 kernel: Lustre: 23678:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 08 22:41:39 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f0c60728d80/0x5d9ee63653edc3aa lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 22 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55cab25de expref: 19 pid: 20555 timeout: 1765959 lvb_type: 0 Jul 08 22:42:40 fir-md1-s1 kernel: LustreError: 10146:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f1dc731fc00 ns: mdt-fir-MDT0002_UUID lock: ffff8f34ea7e0900/0x5d9ee636548d1880 lrc: 3/0,0 mode: PW/PW res: [0x2c002c183:0xb2:0x0].0x0 bits 0x40/0x0 rrc: 10 type: IBT flags: 0x50200000000000 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd57d3b7a expref: 33 pid: 10146 timeout: 0 lvb_type: 0 Jul 08 22:45:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 22:45:17 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 08 22:48:12 fir-md1-s1 kernel: Lustre: 23710:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562651285/real 1562651285] req@ffff8f341ba85700 x1636727211140336/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562651292 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 22:48:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 22:48:30 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 08 22:50:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 22:50:12 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 08 22:51:00 fir-md1-s1 kernel: Lustre: 20727:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1e12182d00 x1631538547465696/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:5/0 lens 480/568 e 0 to 0 dl 1562651465 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 22:51:00 fir-md1-s1 kernel: Lustre: 20727:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 08 22:51:04 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f3332bab600/0x5d9ee63657f89a62 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 22 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55cabe0b5 expref: 19 pid: 21679 timeout: 1766524 lvb_type: 0 Jul 08 22:51:04 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Jul 08 22:54:40 fir-md1-s1 kernel: LustreError: 25087:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.30.23@o2ib6 arrived at 1562651680 with bad export cookie 6746082412932528747 Jul 08 22:55:18 fir-md1-s1 kernel: LustreError: 21268:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.25.23@o2ib6 arrived at 1562651718 with bad export cookie 6746082412868746049 Jul 08 22:55:19 fir-md1-s1 kernel: LustreError: 22288:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f23c7482800 ns: mdt-fir-MDT0002_UUID lock: ffff8f24f8f17bc0/0x5d9ee6365a080b92 lrc: 3/0,0 mode: PW/PW res: [0x2c002c148:0x7d:0x0].0x0 bits 0x40/0x0 rrc: 12 type: IBT flags: 0x50200400000020 nid: 10.8.25.23@o2ib6 remote: 0xeb7608bcd5c0331b expref: 19 pid: 22288 timeout: 0 lvb_type: 0 Jul 08 22:55:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 22:55:43 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 08 22:57:47 fir-md1-s1 kernel: Lustre: 23612:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562651860/real 1562651860] req@ffff8f09946f2d00 x1636727215671536/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562651867 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 22:58:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 22:58:36 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 08 23:00:07 fir-md1-s1 kernel: Lustre: 24577:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1bb0b36c00 x1631776455317904/t0(0) o101->cb1e051f-12ef-c393-c1de-bc60ba01debc@10.8.13.11@o2ib6:11/0 lens 480/568 e 0 to 0 dl 1562652011 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 23:00:07 fir-md1-s1 kernel: Lustre: 24577:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages Jul 08 23:00:11 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.9.103.34@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f204b5a9b00/0x5d9ee6365bd8e59f lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 20 type: IBT flags: 0x60200400000020 nid: 10.9.103.34@o2ib4 remote: 0x479fd480653d26e8 expref: 35 pid: 20511 timeout: 1767071 lvb_type: 0 Jul 08 23:00:11 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Jul 08 23:00:11 fir-md1-s1 kernel: LustreError: 20511:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f2d327f0400 ns: mdt-fir-MDT0002_UUID lock: ffff8f1d60599680/0x5d9ee6365c10038b lrc: 3/0,0 mode: PW/PW res: [0x2c002c148:0x7e:0x0].0x0 bits 0x40/0x0 rrc: 10 type: IBT flags: 0x50200000000000 nid: 10.9.103.34@o2ib4 remote: 0x479fd480653d7799 expref: 11 pid: 20511 timeout: 0 lvb_type: 0 Jul 08 23:00:11 fir-md1-s1 kernel: LustreError: 20511:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages Jul 08 23:00:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 274acbe5-1f09-1bc7-1d04-06ba56c47198 (at 10.8.25.23@o2ib6) reconnecting Jul 08 23:00:14 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 08 23:00:50 fir-md1-s1 kernel: LustreError: 25084:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.13.11@o2ib6 arrived at 1562652050 with bad export cookie 6746082412959469584 Jul 08 23:05:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 23:05:08 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 08 23:06:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 23:06:08 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 08 23:07:17 fir-md1-s1 kernel: Lustre: 20731:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562652430/real 1562652430] req@ffff8f1c4eca8c00 x1636727219171664/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562652437 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 23:08:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 23:08:48 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 08 23:10:03 fir-md1-s1 kernel: Lustre: 20465:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562652596/real 1562652596] req@ffff8f23a172fb00 x1636727220318720/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562652603 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 23:10:27 fir-md1-s1 kernel: Lustre: 23605:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562652620/real 1562652620] req@ffff8f09946f7800 x1636727220492944/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562652627 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 23:10:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 08 23:10:37 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 08 23:12:40 fir-md1-s1 kernel: Lustre: 23745:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562652753/real 1562652753] req@ffff8f34415e6600 x1636727221658160/t0(0) o106->fir-MDT0002@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562652760 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 08 23:12:55 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 08 23:16:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 23:16:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 23:16:55 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 08 23:18:08 fir-md1-s1 kernel: Lustre: 23714:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f34b2cf4500 x1631776456539520/t0(0) o101->cb1e051f-12ef-c393-c1de-bc60ba01debc@10.8.13.11@o2ib6:13/0 lens 480/568 e 1 to 0 dl 1562653093 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 23:18:08 fir-md1-s1 kernel: Lustre: 23714:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Jul 08 23:18:22 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.30.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f34f3317980/0x5d9ee63662fd8af6 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 24 type: IBT flags: 0x60200400000020 nid: 10.8.30.23@o2ib6 remote: 0xe713fe55cafa518 expref: 20 pid: 50584 timeout: 1768162 lvb_type: 0 Jul 08 23:18:22 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jul 08 23:18:46 fir-md1-s1 kernel: LustreError: 31011:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.30.23@o2ib6 arrived at 1562653126 with bad export cookie 6746082412956598821 Jul 08 23:19:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 23:19:07 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jul 08 23:21:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 08 23:21:19 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 08 23:25:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f474a000, cur 1562653546 expire 1562653396 last 1562653319 Jul 08 23:27:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 23:27:13 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 08 23:29:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 23:29:24 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 08 23:31:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 08 23:31:26 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 08 23:33:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 23:33:49 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 08 23:37:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 23:37:29 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 08 23:39:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 08 23:39:25 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 08 23:41:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 08 23:41:39 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 08 23:47:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 23:47:35 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 08 23:49:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 08 23:49:27 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 08 23:50:28 fir-md1-s1 kernel: Lustre: 22429:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0c050bb850 x1631600503280448/t0(0) o3->657250be-d5db-acec-954e-1239d7463eca@10.9.104.65@o2ib4:2/0 lens 488/8632 e 1 to 0 dl 1562655032 ref 2 fl Interpret:/0/0 rc 0/0 Jul 08 23:50:28 fir-md1-s1 kernel: Lustre: 22429:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 08 23:51:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 08 23:51:20 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 08 23:52:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 08 23:52:16 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 08 23:57:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 08 23:57:41 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 09 00:00:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 00:00:15 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 09 00:02:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 00:02:25 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 00:03:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 00:03:12 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 00:07:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 00:07:47 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 00:11:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 00:11:34 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 09 00:12:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 00:12:28 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 09 00:15:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 00:15:01 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 00:17:46 fir-md1-s1 kernel: Lustre: 21485:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0c050bd850 x1634176625220320/t0(0) o3->bff671a6-6393-a53b-8c2a-0f521cd0a513@10.9.109.13@o2ib4:21/0 lens 488/16824 e 1 to 0 dl 1562656671 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:17:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 00:17:53 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 09 00:21:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 00:21:36 fir-md1-s1 kernel: Lustre: Skipped 104 previous similar messages Jul 09 00:23:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 00:23:48 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 09 00:28:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 00:28:08 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 00:28:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 00:28:41 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 09 00:31:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 00:31:37 fir-md1-s1 kernel: Lustre: Skipped 103 previous similar messages Jul 09 00:32:11 fir-md1-s1 kernel: Lustre: 21433:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f204858f800 x1631686395367168/t0(0) o101->8a2377b9-dd4d-1468-124f-a22e5b47b9b4@10.8.11.23@o2ib6:15/0 lens 376/1600 e 0 to 0 dl 1562657535 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:33:06 fir-md1-s1 kernel: Lustre: 22288:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562657579/real 1562657579] req@ffff8f1eddda4b00 x1636727371386944/t0(0) o106->fir-MDT0002@10.8.22.20@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562657586 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 09 00:33:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 26ac517c-0ccc-5f83-6680-5e234583a053 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d3c036400, cur 1562657628 expire 1562657478 last 1562657401 Jul 09 00:33:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 00:33:57 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 09 00:34:23 fir-md1-s1 kernel: Lustre: 23713:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f36eef19200 x1631538592399248/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:28/0 lens 480/568 e 1 to 0 dl 1562657668 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:34:54 fir-md1-s1 kernel: Lustre: 97670:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1af46e6000 x1631779714566112/t0(0) o101->a5959e71-bc10-93fe-ec09-fd083077a83e@10.8.24.26@o2ib6:29/0 lens 480/568 e 0 to 0 dl 1562657699 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:35:38 fir-md1-s1 kernel: LustreError: 50581:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562657648, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1dc0abc140/0x5d9ee636844156a6 lrc: 3/0,1 mode: --/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 21 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 50581 timeout: 0 lvb_type: 0 Jul 09 00:36:16 fir-md1-s1 kernel: Lustre: 21333:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f26612f3300 x1631779714621024/t0(0) o101->a5959e71-bc10-93fe-ec09-fd083077a83e@10.8.24.26@o2ib6:21/0 lens 480/568 e 0 to 0 dl 1562657781 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:36:20 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.22.20@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f20c6e82400/0x5d9ee636866668f2 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 22 type: IBT flags: 0x60200400000020 nid: 10.8.22.20@o2ib6 remote: 0xe96edf08a2c73ad9 expref: 44 pid: 21482 timeout: 1772840 lvb_type: 0 Jul 09 00:36:30 fir-md1-s1 kernel: Lustre: 23634:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562657783/real 1562657783] req@ffff8f1261da2400 x1636727472516928/t0(0) o106->fir-MDT0002@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562657790 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 09 00:36:30 fir-md1-s1 kernel: Lustre: 23635:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f450e784500 x1631538593904768/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:5/0 lens 480/568 e 0 to 0 dl 1562657795 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:36:37 fir-md1-s1 kernel: Lustre: 23634:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562657790/real 1562657790] req@ffff8f1261da2400 x1636727472516928/t0(0) o106->fir-MDT0002@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562657797 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 09 00:38:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 00:38:58 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 09 00:40:59 fir-md1-s1 kernel: Lustre: 21452:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562658051/real 1562658051] req@ffff8f27997d0c00 x1636727498629504/t0(0) o104->fir-MDT0002@10.8.11.6@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562658058 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 09 00:41:10 fir-md1-s1 kernel: Lustre: 23748:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2692228f00 x1631755762631056/t0(0) o101->6102ee9c-599d-0d29-7336-fa30c59b9711@10.8.20.10@o2ib6:15/0 lens 480/568 e 0 to 0 dl 1562658075 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:41:14 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.11.6@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2893fa72c0/0x5d9ee636897d28d3 lrc: 3/0,0 mode: PW/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 17 type: IBT flags: 0x60200400000020 nid: 10.8.11.6@o2ib6 remote: 0x721c85a22645f217 expref: 45 pid: 21333 timeout: 1773134 lvb_type: 0 Jul 09 00:41:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 00:41:42 fir-md1-s1 kernel: Lustre: Skipped 108 previous similar messages Jul 09 00:41:49 fir-md1-s1 kernel: Lustre: 20555:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f19f28b6900 x1631538615196608/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:24/0 lens 480/568 e 0 to 0 dl 1562658114 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:43:08 fir-md1-s1 kernel: Lustre: 20511:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f22efd9bf00 x1631779727080784/t0(0) o101->a5959e71-bc10-93fe-ec09-fd083077a83e@10.8.24.26@o2ib6:13/0 lens 480/568 e 0 to 0 dl 1562658193 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:43:08 fir-md1-s1 kernel: Lustre: 20511:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 09 00:44:17 fir-md1-s1 kernel: LNetError: 20186:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jul 09 00:44:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 00:44:25 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 09 00:45:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 00:45:28 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 00:46:07 fir-md1-s1 kernel: Lustre: 23713:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/3), not sending early reply req@ffff8f4392b54200 x1631538634146480/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:12/0 lens 480/568 e 0 to 0 dl 1562658372 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:46:07 fir-md1-s1 kernel: Lustre: 23713:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 09 00:46:46 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.24.26@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1a48f31680/0x5d9ee6368cbb73d3 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 19 type: IBT flags: 0x60200400000020 nid: 10.8.24.26@o2ib6 remote: 0x532ae402ed14dc60 expref: 19 pid: 23733 timeout: 1773466 lvb_type: 0 Jul 09 00:46:46 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Jul 09 00:47:10 fir-md1-s1 kernel: LustreError: 48115:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.24.26@o2ib6 arrived at 1562658430 with bad export cookie 6746082413791205776 Jul 09 00:50:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 00:50:37 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 09 00:51:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 00:51:53 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 09 00:52:14 fir-md1-s1 kernel: Lustre: 23455:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1f4ed2f200 x1631686418353328/t0(0) o101->8a2377b9-dd4d-1468-124f-a22e5b47b9b4@10.8.11.23@o2ib6:19/0 lens 480/568 e 0 to 0 dl 1562658739 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 00:52:14 fir-md1-s1 kernel: Lustre: 23455:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages Jul 09 00:52:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6da12e78-70c3-9109-6c3f-cc3cd573cc58 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f28323dd800, cur 1562658777 expire 1562658627 last 1562658550 Jul 09 00:52:57 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 00:54:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 00:54:36 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 00:55:07 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.9.103.34@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f267e75bcc0/0x5d9ee636920d05bc lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 20 type: IBT flags: 0x60200400000020 nid: 10.9.103.34@o2ib4 remote: 0x479fd480673e7ecd expref: 258 pid: 23608 timeout: 1773967 lvb_type: 0 Jul 09 00:55:07 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Jul 09 00:57:08 fir-md1-s1 kernel: LustreError: 23103:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.24.26@o2ib6 arrived at 1562659028 with bad export cookie 6746082413824618848 Jul 09 00:57:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 00:57:17 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 01:00:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 01:00:52 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 09 01:01:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 01:01:53 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 09 01:04:12 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f26a28e6300 x1631686435373952/t0(0) o101->8a2377b9-dd4d-1468-124f-a22e5b47b9b4@10.8.11.23@o2ib6:17/0 lens 480/568 e 0 to 0 dl 1562659457 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 01:04:12 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages Jul 09 01:04:28 fir-md1-s1 kernel: Lustre: 23644:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562659461/real 1562659461] req@ffff8f10a97f9500 x1636727615415392/t0(0) o106->fir-MDT0002@10.8.22.20@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562659468 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 09 01:04:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 01:04:49 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 01:05:17 fir-md1-s1 kernel: LustreError: 23645:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562659427, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1dab428240/0x5d9ee6369821c9b1 lrc: 3/0,1 mode: --/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 17 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 23645 timeout: 0 lvb_type: 0 Jul 09 01:05:17 fir-md1-s1 kernel: LustreError: 23645:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 09 01:05:31 fir-md1-s1 kernel: Lustre: 23652:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562659524/real 1562659524] req@ffff8f2ddae99800 x1636727620559008/t0(0) o106->fir-MDT0002@10.8.22.20@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562659531 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 09 01:06:10 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.11.23@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1dab428240/0x5d9ee6369821c9b1 lrc: 3/0,0 mode: PW/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 17 type: IBT flags: 0x60200400000020 nid: 10.8.11.23@o2ib6 remote: 0x685a2eace538c518 expref: 19 pid: 23645 timeout: 1774630 lvb_type: 0 Jul 09 01:06:10 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages Jul 09 01:06:23 fir-md1-s1 kernel: Lustre: 23716:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562659576/real 1562659576] req@ffff8f26a63edd00 x1636727624842192/t0(0) o106->fir-MDT0002@10.8.22.20@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562659583 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 09 01:09:10 fir-md1-s1 kernel: LustreError: 97660:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562659660, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1e46f3dc40/0x5d9ee6369a847bdb lrc: 3/0,1 mode: --/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 12 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97660 timeout: 0 lvb_type: 0 Jul 09 01:09:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 01:09:47 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 01:11:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 01:11:53 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 09 01:12:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 09 01:12:20 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 09 01:14:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 01:14:51 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 09 01:17:26 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2798a27800 x1631779760667632/t0(0) o101->a5959e71-bc10-93fe-ec09-fd083077a83e@10.8.24.26@o2ib6:1/0 lens 480/568 e 0 to 0 dl 1562660251 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 01:17:26 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 15 previous similar messages Jul 09 01:20:35 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.11.6@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2022bb0000/0x5d9ee636a1f3fbe3 lrc: 3/0,0 mode: PW/PW res: [0x2c002c180:0x7c:0x0].0x0 bits 0x40/0x0 rrc: 17 type: IBT flags: 0x60200400000020 nid: 10.8.11.6@o2ib6 remote: 0x721c85a226478ed1 expref: 20 pid: 97672 timeout: 1775495 lvb_type: 0 Jul 09 01:20:35 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Jul 09 01:20:40 fir-md1-s1 kernel: LustreError: 25085:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.11.6@o2ib6 arrived at 1562660440 with bad export cookie 6746082414046326733 Jul 09 01:22:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 01:22:17 fir-md1-s1 kernel: Lustre: Skipped 116 previous similar messages Jul 09 01:22:52 fir-md1-s1 kernel: Lustre: 21333:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562660565/real 1562660565] req@ffff8f2fc6dee000 x1636727696927184/t0(0) o106->fir-MDT0002@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562660572 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 09 01:22:59 fir-md1-s1 kernel: Lustre: 97655:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562660572/real 1562660572] req@ffff8f1e35fc0c00 x1636727697331696/t0(0) o106->fir-MDT0002@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562660579 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 09 01:23:06 fir-md1-s1 kernel: Lustre: 97655:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562660579/real 1562660579] req@ffff8f1e35fc0c00 x1636727697331696/t0(0) o106->fir-MDT0002@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562660586 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 09 01:23:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 01:23:12 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 09 01:25:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 01:25:19 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 09 01:27:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 01:27:26 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 01:27:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 0ebe3954-8665-c753-62ab-a40297bf966d (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2b0a2fec00, cur 1562660848 expire 1562660698 last 1562660621 Jul 09 01:27:28 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 01:30:16 fir-md1-s1 kernel: Lustre: 23754:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2622fc0c00 x1631538751564720/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:21/0 lens 480/568 e 0 to 0 dl 1562661021 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 01:30:16 fir-md1-s1 kernel: Lustre: 23754:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages Jul 09 01:32:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 01:32:20 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 09 01:32:49 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.9.103.34@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f2ae8f6de80/0x5d9ee636a85e7321 lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x4:0x0].0x0 bits 0x40/0x0 rrc: 15 type: IBT flags: 0x60200400000020 nid: 10.9.103.34@o2ib4 remote: 0x479fd480690b6545 expref: 26 pid: 23692 timeout: 1776229 lvb_type: 0 Jul 09 01:32:49 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Jul 09 01:33:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.7.29@o2ib6, removing former export from same NID Jul 09 01:33:45 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 01:35:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 01:35:36 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 09 01:39:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 01:39:30 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 09 01:42:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 01:42:24 fir-md1-s1 kernel: Lustre: Skipped 134 previous similar messages Jul 09 01:44:36 fir-md1-s1 kernel: Lustre: 21456:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f202ae13600 x1631538758613536/t0(0) o101->d3a33565-cf5d-2ffd-ba04-f0bdcb5e77d8@10.9.103.34@o2ib4:10/0 lens 480/568 e 0 to 0 dl 1562661880 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 01:44:36 fir-md1-s1 kernel: Lustre: 21456:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages Jul 09 01:44:39 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.22.20@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f26e0f2ba80/0x5d9ee636ad2aa44c lrc: 3/0,0 mode: PW/PW res: [0x2c002c409:0x3:0x0].0x0 bits 0x40/0x0 rrc: 15 type: IBT flags: 0x60200400000020 nid: 10.8.22.20@o2ib6 remote: 0xe96edf08a2ce36bf expref: 19 pid: 23747 timeout: 1776939 lvb_type: 0 Jul 09 01:44:39 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages Jul 09 01:45:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.8.2@o2ib6, removing former export from same NID Jul 09 01:45:18 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 09 01:45:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 01:45:57 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 09 01:49:00 fir-md1-s1 kernel: LustreError: 21765:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.22.20@o2ib6 arrived at 1562662140 with bad export cookie 6746082414362191954 Jul 09 01:51:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 01:51:20 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 01:52:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 01:52:35 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 09 01:55:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 01:55:50 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 09 01:56:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 01:56:01 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 01:56:13 fir-md1-s1 kernel: Lustre: 22279:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f206aac7200 x1634160005722400/t0(0) o101->32315fe6-6915-bd82-691a-5460d13ab6db@10.9.103.27@o2ib4:18/0 lens 480/568 e 0 to 0 dl 1562662578 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 01:56:13 fir-md1-s1 kernel: Lustre: 22279:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jul 09 01:56:17 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.11.6@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f17dd6f5340/0x5d9ee636b1dc16ac lrc: 3/0,0 mode: PW/PW res: [0x2c002c180:0x7c:0x0].0x0 bits 0x40/0x0 rrc: 23 type: IBT flags: 0x60200400000020 nid: 10.8.11.6@o2ib6 remote: 0x721c85a22648110e expref: 20 pid: 24577 timeout: 1777637 lvb_type: 0 Jul 09 01:56:17 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jul 09 01:56:56 fir-md1-s1 kernel: LustreError: 25028:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.11.23@o2ib6 arrived at 1562662616 with bad export cookie 6746082414329937900 Jul 09 02:00:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client b30c01a7-931a-8263-f304-966fa9bd47ec (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1dbc608000, cur 1562662817 expire 1562662667 last 1562662590 Jul 09 02:00:17 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 02:03:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 02:03:01 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 09 02:06:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 02:06:04 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 09 02:06:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 02:06:06 fir-md1-s1 kernel: Lustre: Skipped 163427 previous similar messages Jul 09 02:07:19 fir-md1-s1 kernel: Lustre: 23747:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f34b9f7c200 x1631686488015440/t0(0) o101->8a2377b9-dd4d-1468-124f-a22e5b47b9b4@10.8.11.23@o2ib6:24/0 lens 480/568 e 0 to 0 dl 1562663244 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 02:07:19 fir-md1-s1 kernel: Lustre: 23747:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages Jul 09 02:07:24 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.20.10@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1ccba87980/0x5d9ee636b61779f9 lrc: 3/0,0 mode: PW/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 26 type: IBT flags: 0x60200400000020 nid: 10.8.20.10@o2ib6 remote: 0x1f0dec44cd243bfd expref: 19 pid: 97660 timeout: 1778304 lvb_type: 0 Jul 09 02:07:24 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jul 09 02:08:03 fir-md1-s1 kernel: LustreError: 31007:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.11.23@o2ib6 arrived at 1562663283 with bad export cookie 6746082414442404366 Jul 09 02:09:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 4a34d9ca-85d3-d986-1b27-304345ee5afb (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2fb6df2000, cur 1562663380 expire 1562663230 last 1562663153 Jul 09 02:09:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 02:09:56 fir-md1-s1 kernel: LustreError: 20369:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.25.23@o2ib6 arrived at 1562663396 with bad export cookie 6746082412960696376 Jul 09 02:10:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 02:10:05 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 02:10:28 fir-md1-s1 kernel: LustreError: 20555:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562663337, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1e55e99440/0x5d9ee636b6cab2df lrc: 3/0,1 mode: --/PW res: [0x2c002c180:0x7c:0x0].0x0 bits 0x40/0x0 rrc: 13 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20555 timeout: 0 lvb_type: 0 Jul 09 02:13:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 02:13:03 fir-md1-s1 kernel: Lustre: Skipped 163418 previous similar messages Jul 09 02:13:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client b7caf93d-2daa-26e6-33b8-897c7ea93dd8 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f186cfcf800, cur 1562663621 expire 1562663471 last 1562663394 Jul 09 02:13:41 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 02:13:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client b7caf93d-2daa-26e6-33b8-897c7ea93dd8 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1af16af000, cur 1562663638 expire 1562663488 last 1562663411 Jul 09 02:14:41 fir-md1-s1 kernel: LustreError: 24580:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562663591, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1a7ba3e0c0/0x5d9ee636b8308d2f lrc: 3/0,1 mode: --/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 27 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 24580 timeout: 0 lvb_type: 0 Jul 09 02:16:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6102ee9c-599d-0d29-7336-fa30c59b9711 (at 10.8.20.10@o2ib6) reconnecting Jul 09 02:16:09 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 02:16:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 02:16:13 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 09 02:16:51 fir-md1-s1 kernel: LustreError: 97638:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562663721, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1638c9d580/0x5d9ee636b8e525a4 lrc: 3/0,1 mode: --/PW res: [0x2c002c180:0x7c:0x0].0x0 bits 0x40/0x0 rrc: 12 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97638 timeout: 0 lvb_type: 0 Jul 09 02:16:51 fir-md1-s1 kernel: LustreError: 97638:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages Jul 09 02:17:02 fir-md1-s1 kernel: LustreError: 25086:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.20.10@o2ib6 arrived at 1562663822 with bad export cookie 6746082414526375925 Jul 09 02:21:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1c4743a800, cur 1562664068 expire 1562663918 last 1562663841 Jul 09 02:21:08 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 09 02:23:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 02:23:29 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 09 02:25:39 fir-md1-s1 kernel: Lustre: 23747:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f3478e20c00 x1633783776209328/t0(0) o101->274acbe5-1f09-1bc7-1d04-06ba56c47198@10.8.25.23@o2ib6:14/0 lens 480/568 e 0 to 0 dl 1562664344 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 02:25:39 fir-md1-s1 kernel: Lustre: 23747:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 14 previous similar messages Jul 09 02:26:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 02:26:14 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 02:26:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 8a2377b9-dd4d-1468-124f-a22e5b47b9b4 (at 10.8.11.23@o2ib6) reconnecting Jul 09 02:26:14 fir-md1-s1 kernel: Lustre: Skipped 153227 previous similar messages Jul 09 02:26:45 fir-md1-s1 kernel: LustreError: 23748:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562664314, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f324ab2e540/0x5d9ee636bc272085 lrc: 3/0,1 mode: --/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 27 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 23748 timeout: 0 lvb_type: 0 Jul 09 02:26:59 fir-md1-s1 kernel: LustreError: 22288:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562664329, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f1bba081440/0x5d9ee636bc384037 lrc: 3/0,1 mode: --/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 27 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 22288 timeout: 0 lvb_type: 0 Jul 09 02:26:59 fir-md1-s1 kernel: LustreError: 22288:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 09 02:27:13 fir-md1-s1 kernel: LustreError: 97643:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562664343, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f171500f980/0x5d9ee636bc48eb3c lrc: 3/0,1 mode: --/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 27 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97643 timeout: 0 lvb_type: 0 Jul 09 02:27:44 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.8.11.6@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2b6c640900/0x5d9ee636bc270a51 lrc: 3/0,0 mode: PW/PW res: [0x2c002c180:0x7b:0x0].0x0 bits 0x40/0x0 rrc: 27 type: IBT flags: 0x60200400000020 nid: 10.8.11.6@o2ib6 remote: 0x721c85a226485571 expref: 14 pid: 23748 timeout: 1779524 lvb_type: 0 Jul 09 02:27:44 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 9 previous similar messages Jul 09 02:27:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client b7eb93d5-8c42-223b-054b-48b7832859bc (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f16d5042000, cur 1562664478 expire 1562664328 last 1562664251 Jul 09 02:31:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 02:31:18 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 02:33:41 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 02:33:41 fir-md1-s1 kernel: Lustre: Skipped 153278 previous similar messages Jul 09 02:35:56 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 188b3bd1-23a6-543f-00d7-3c05d963cb64 (at 10.8.11.9@o2ib6) in 153 seconds. I think it's dead, and I am evicting it. exp ffff8f439c6b6400, cur 1562664956 expire 1562664806 last 1562664803 Jul 09 02:35:56 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 02:36:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 02:36:17 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 02:36:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 02:36:17 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 09 02:37:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 12cdaed2-086d-f211-b5e6-a7a51b57bbf6 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f247b6eec00, cur 1562665030 expire 1562664880 last 1562664803 Jul 09 02:40:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 02:41:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9df2cc07-ba94-1ea2-6172-f47b09f55c82 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f167f389400, cur 1562665270 expire 1562665120 last 1562665043 Jul 09 02:41:10 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 09 02:43:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 02:43:53 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 09 02:45:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client f691ce56-c75a-3453-35b5-9cac0a6f187c (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2e0a998800, cur 1562665512 expire 1562665362 last 1562665285 Jul 09 02:45:12 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 02:46:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 02:46:18 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 02:46:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 02:46:20 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 09 02:50:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 02:53:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 02:53:54 fir-md1-s1 kernel: Lustre: Skipped 102 previous similar messages Jul 09 02:54:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e2e8b6fe-9a67-1617-a235-c6cc38ba57d4 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34f3567800, cur 1562666055 expire 1562665905 last 1562665828 Jul 09 02:54:15 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 02:57:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 02:57:14 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 09 02:57:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 02:57:16 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 09 03:03:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 03:03:55 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 09 03:07:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 03:07:23 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 09 03:09:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 03:09:06 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 09 03:10:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 03:14:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 03:14:00 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 09 03:14:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 03:14:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d4704d07-4d9d-83e2-a0bd-ed6cd3778ee5 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f3629400, cur 1562667294 expire 1562667144 last 1562667067 Jul 09 03:14:54 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 09 03:17:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 03:18:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 03:18:05 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 03:19:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 03:19:07 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 09 03:20:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 3826547f-d431-1d44-4311-1be321d906e4 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f189d0a0400, cur 1562667612 expire 1562667462 last 1562667385 Jul 09 03:20:12 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 03:20:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 03:20:35 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 03:24:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 03:24:13 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 09 03:28:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 03:28:11 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 03:29:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 03:29:30 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 03:33:40 fir-md1-s1 kernel: LNetError: 20196:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jul 09 03:34:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 03:34:17 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 09 03:34:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 03:38:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 03:38:40 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 03:40:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f279a64b000, cur 1562668842 expire 1562668692 last 1562668615 Jul 09 03:40:42 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 03:40:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 03:40:57 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 03:44:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 03:44:22 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 09 03:46:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 03:46:19 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 03:48:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5bf7a607-5118-27c6-615a-5015949857b5 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2383815800, cur 1562669318 expire 1562669168 last 1562669091 Jul 09 03:49:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 03:49:42 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 09 03:52:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 03:52:26 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 09 03:54:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 03:54:25 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 09 04:00:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 04:00:07 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 09 04:00:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 04:00:51 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 04:03:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 04:03:36 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 09 04:05:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 04:05:00 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 09 04:11:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 04:11:01 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 04:13:13 fir-md1-s1 kernel: LNetError: 20183:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 09 04:13:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 04:13:57 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 04:15:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 04:15:02 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 09 04:16:34 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 04:16:34 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 04:22:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 04:22:29 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 04:25:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 04:25:13 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 09 04:25:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 04:25:13 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 09 04:32:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 04:32:37 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 09 04:34:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 04:34:00 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 04:36:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 04468791-317f-0b85-a724-e5fbf6594482 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2672b24400, cur 1562672169 expire 1562672019 last 1562671942 Jul 09 04:36:09 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 04:36:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 04:36:16 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 09 04:37:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 04:37:03 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 09 04:44:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 04:44:23 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 09 04:46:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 04:46:18 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 04:47:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 09 04:47:28 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 04:48:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 04:48:52 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 04:49:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c0143168-7b00-5187-33ee-2ee23ada0e35 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2826e09c00, cur 1562672946 expire 1562672796 last 1562672719 Jul 09 04:49:06 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 04:54:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 04:54:55 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 04:56:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 04:56:23 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 09 04:57:59 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 04:57:59 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 09 04:59:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 04:59:17 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 05:05:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 05:05:05 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 09 05:06:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 05:06:48 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 05:08:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 05:08:01 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 05:15:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 05:15:14 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 05:17:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 05:17:00 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 09 05:19:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 05:19:07 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 09 05:25:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 05:25:17 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 09 05:27:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 05:27:34 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 09 05:28:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 05:29:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 05:29:08 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 09 05:30:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 05:36:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 05:36:16 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 09 05:37:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 05:37:36 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 09 05:40:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 05:40:17 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 05:46:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 05:46:19 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 05:46:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 05:46:52 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 05:47:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 05:47:44 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 09 05:50:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 05:50:56 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 09 05:52:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 05:56:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 05:56:25 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 09 05:57:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 05:57:47 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 09 06:01:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 06:01:27 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 06:06:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 06:06:30 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 06:07:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 06:07:53 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 09 06:11:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 06:11:27 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 09 06:15:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:17:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 06:17:00 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 06:17:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:17:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 06:17:57 fir-md1-s1 kernel: Lustre: Skipped 108 previous similar messages Jul 09 06:18:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 1103d677-bcdc-c647-1248-807c12ba22a8 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f17bcc02c00, cur 1562678290 expire 1562678140 last 1562678063 Jul 09 06:18:10 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 06:19:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:22:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 06:22:31 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 09 06:27:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 06:27:00 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 09 06:27:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 06:27:58 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 09 06:32:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 06:32:32 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 06:35:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 3d028c2f-2477-2a00-2f10-1e73838f7457 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f347ed00800, cur 1562679326 expire 1562679176 last 1562679099 Jul 09 06:35:26 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 06:36:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:37:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 06:37:10 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 06:37:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:38:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 06:38:03 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 09 06:38:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:40:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:40:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 94cb6da7-d582-1b33-0e0e-34207c65c599 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2fc6deec00, cur 1562679648 expire 1562679498 last 1562679421 Jul 09 06:40:48 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 06:42:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) in 207 seconds. I think it's dead, and I am evicting it. exp ffff8f34ec838800, cur 1562679724 expire 1562679574 last 1562679517 Jul 09 06:42:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 06:42:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 06:42:36 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 09 06:47:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 06:47:17 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 09 06:48:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 06:48:12 fir-md1-s1 kernel: Lustre: Skipped 106 previous similar messages Jul 09 06:48:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:53:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 06:53:57 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 09 06:57:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 06:57:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 06:57:21 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 09 06:58:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 06:58:16 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 09 06:58:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 07:03:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 4a775cba-723d-a68b-1ff4-ae110efb02b5 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f171e1e2000, cur 1562681039 expire 1562680889 last 1562680812 Jul 09 07:04:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 07:04:05 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 07:04:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 07:04:52 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 07:07:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 07:07:47 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 07:08:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 07:08:18 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 09 07:09:04 fir-md1-s1 kernel: Lustre: 23631:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f379c388000 x1636571295304848/t0(0) o101->86fa2497-cbd1-3103-4628-e12187b558d9@10.9.101.25@o2ib4:9/0 lens 480/568 e 1 to 0 dl 1562681349 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 07:09:04 fir-md1-s1 kernel: Lustre: 23631:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages Jul 09 07:11:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 07:14:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 07:14:33 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 09 07:18:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 07:18:28 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 07:18:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 07:18:28 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 09 07:24:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 07:24:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 07:24:35 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 07:24:35 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 09 07:29:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 07:29:35 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 07:29:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 07:29:35 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 09 07:35:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 07:35:12 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 09 07:35:12 fir-md1-s1 kernel: Lustre: 22286:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-1), not sending early reply req@ffff8f1d6abe6000 x1635107050306688/t0(0) o101->83887939-6757-4aea-8b88-f0aa38eb91bc@10.9.108.13@o2ib4:17/0 lens 576/3264 e 0 to 0 dl 1562682917 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 07:35:12 fir-md1-s1 kernel: Lustre: 22286:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages Jul 09 07:35:22 fir-md1-s1 kernel: Lustre: 26253:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-1), not sending early reply req@ffff8f1dee52f500 x1635089897588224/t0(0) o101->2c084bd6-6132-6737-34f2-02b28f3edaf8@10.9.109.32@o2ib4:27/0 lens 576/0 e 0 to 0 dl 1562682927 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 09 07:35:22 fir-md1-s1 kernel: Lustre: 26253:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1059 previous similar messages Jul 09 07:35:25 fir-md1-s1 kernel: Lustre: 23585:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (27:1s); client may timeout. req@ffff8f07da244200 x1638234212823008/t0(0) o101->820a82e4-064a-d399-a663-1803c58bca77@10.9.112.15@o2ib4:24/0 lens 576/592 e 0 to 0 dl 1562682924 ref 1 fl Complete:/0/0 rc 0/0 Jul 09 07:35:25 fir-md1-s1 kernel: LustreError: 21410:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.102.20@o2ib4: deadline 27:1s ago req@ffff8f1338bbd100 x1631568602423920/t0(0) o101->0db2d4e0-bf1e-3689-817d-00b10dcb4858@10.9.102.20@o2ib4:24/0 lens 576/0 e 0 to 0 dl 1562682924 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 09 07:35:25 fir-md1-s1 kernel: LustreError: 21410:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 12 previous similar messages Jul 09 07:35:25 fir-md1-s1 kernel: Lustre: 23585:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 317 previous similar messages Jul 09 07:38:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 07:38:30 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 07:39:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 07:39:36 fir-md1-s1 kernel: Lustre: Skipped 560 previous similar messages Jul 09 07:39:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 07:39:39 fir-md1-s1 kernel: Lustre: Skipped 503 previous similar messages Jul 09 07:47:12 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 07:47:12 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 09 07:47:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 07:47:18 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 09 07:47:19 fir-md1-s1 kernel: Lustre: 21446:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f22e2333f00 x1631660325794608/t0(0) o101->b7aae4ae-1aa0-9e5d-5ecf-90e4dbcd33de@10.9.101.27@o2ib4:24/0 lens 480/568 e 1 to 0 dl 1562683644 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 07:47:19 fir-md1-s1 kernel: Lustre: 21446:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 171 previous similar messages Jul 09 07:47:22 fir-md1-s1 kernel: Lustre: 23759:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2f91a1fb00 x1638235634307120/t0(0) o101->8effb155-901a-a135-30ea-62c11eaaf5e4@10.9.101.55@o2ib4:27/0 lens 480/568 e 1 to 0 dl 1562683647 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 07:47:22 fir-md1-s1 kernel: Lustre: 23759:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jul 09 07:47:28 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 07:47:32 fir-md1-s1 kernel: Lustre: 23077:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2d32646900 x1633919152572352/t0(0) o101->b731aa74-f761-f808-ac4e-60997bf2bd97@10.9.101.51@o2ib4:7/0 lens 480/568 e 0 to 0 dl 1562683657 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 07:47:32 fir-md1-s1 kernel: Lustre: 23077:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Jul 09 07:47:33 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 07:47:41 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 07:47:54 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 07:48:02 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 07:49:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 07:49:57 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 07:50:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 07:50:01 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 07:50:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 07:50:22 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 07:51:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 83681611-079a-5f4e-8864-a59fd70f2c12 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2b5ef8f400, cur 1562683891 expire 1562683741 last 1562683664 Jul 09 07:51:31 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 07:51:33 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 07:51:33 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 09 07:51:42 fir-md1-s1 kernel: Lustre: 21680:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f0e65a21800 x1634132356673712/t0(0) o101->05133d08-3c30-bc0b-3005-cf52634e4b28@10.9.101.47@o2ib4:17/0 lens 480/568 e 0 to 0 dl 1562683907 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 07:52:09 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 07:52:09 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 09 07:55:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fc3fb400, cur 1562684109 expire 1562683959 last 1562683882 Jul 09 07:55:09 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 07:57:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 07:57:26 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 09 08:00:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 08:00:02 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 09 08:00:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 08:00:31 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 09 08:09:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 08:09:39 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 09 08:10:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 08:10:29 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 09 08:10:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 08:10:55 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 08:13:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 08:13:23 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 08:18:01 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3e04e13800, cur 1562685481 expire 1562685331 last 1562685254 Jul 09 08:18:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 08:18:38 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 08:21:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 08:21:03 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 09 08:21:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 08:21:03 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 09 08:21:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 08:21:07 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 08:24:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 08:30:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 08:30:57 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 08:31:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 08:31:07 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 08:31:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 08:31:07 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 09 08:31:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 08:31:11 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 08:38:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f290631c000, cur 1562686693 expire 1562686543 last 1562686466 Jul 09 08:41:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 08:41:05 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 08:41:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 08:41:12 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 08:41:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 08:41:12 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 09 08:41:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 907cc100-42d3-4f58-47b7-3f525e8fafee (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1ecfc62400, cur 1562686886 expire 1562686736 last 1562686659 Jul 09 08:41:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 08:41:58 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 09 08:51:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 08:51:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 08:51:21 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 08:51:21 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 09 08:53:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 08:53:08 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 08:54:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 08:54:58 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 09:01:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 09:01:37 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 09 09:03:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 09:03:36 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 09 09:03:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 09:03:38 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 09:11:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 09:11:48 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 09:11:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 09:11:53 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 09 09:13:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 09:13:42 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 09:14:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 09:14:05 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 09:21:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 09:21:58 fir-md1-s1 kernel: Lustre: Skipped 108 previous similar messages Jul 09 09:24:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 09:24:08 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 09 09:24:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 09:24:13 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 09:26:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 09:26:50 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 09:32:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 09:32:18 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 09 09:34:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 09:34:23 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 09:35:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 09 09:35:31 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 09:36:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 09:36:50 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 09 09:42:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 09:42:25 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 09 09:45:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 09:45:22 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 09:45:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 09:45:53 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 09:48:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 09:48:04 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 09 09:52:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 09:52:32 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 09 09:55:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 09:55:32 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 09 09:57:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 09:57:33 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 10:01:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 10:01:22 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 09 10:03:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 10:03:00 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 09 10:05:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 10:05:46 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 10:07:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 10:07:42 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 10:11:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 10:11:25 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 09 10:13:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 10:13:36 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 09 10:16:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 10:16:11 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 10:18:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 10:18:26 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 10:23:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 10:23:37 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 09 10:24:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 10:24:36 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 09 10:26:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 10:26:12 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 10:28:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 10:28:47 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 09 10:34:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 10:34:16 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 09 10:36:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 10:36:18 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 10:36:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 10:36:37 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 10:40:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 10:40:19 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 10:40:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f266a9c7400, cur 1562694042 expire 1562693892 last 1562693815 Jul 09 10:40:42 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 10:44:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 10:44:16 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 09 10:46:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 10:46:28 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 09 10:46:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 10:46:49 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 10:50:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 10:50:43 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 09 10:54:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 10:54:52 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 09 10:56:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 10:56:32 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 10:59:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 10:59:15 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 11:00:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 11:00:45 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 09 11:04:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 11:04:58 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 09 11:06:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 11:06:39 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 11:10:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 11:10:46 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 09 11:12:45 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 11:12:45 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 09 11:12:54 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 11:12:54 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 18 previous similar messages Jul 09 11:13:11 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 09 11:13:11 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22 previous similar messages Jul 09 11:13:44 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 09 11:13:44 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 64 previous similar messages Jul 09 11:14:49 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 11:14:49 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 146 previous similar messages Jul 09 11:14:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 11:14:58 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 09 11:16:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 11:16:54 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 11:16:58 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 11:16:58 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 287 previous similar messages Jul 09 11:17:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 11:17:28 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 11:20:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 11:20:53 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 09 11:21:15 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 11:21:15 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 610 previous similar messages Jul 09 11:25:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 11:25:02 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 09 11:27:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 11:27:00 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 09 11:29:52 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 11:29:52 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1230 previous similar messages Jul 09 11:30:59 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 11:30:59 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 09 11:31:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 11:31:47 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 11:35:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 11:35:02 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 09 11:37:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 11:37:04 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 11:39:52 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 155648 GRANT, real grant 0 Jul 09 11:39:52 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2374 previous similar messages Jul 09 11:41:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 11:41:07 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 09 11:45:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 11:45:07 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 09 11:45:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 11:45:38 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 11:49:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 11:49:06 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 11:49:53 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 09 11:49:53 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1286 previous similar messages Jul 09 11:51:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 11:51:32 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 09 11:55:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 11:55:13 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 09 11:55:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 11:55:45 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 11:59:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 11:59:47 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 11:59:55 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 11:59:55 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1188 previous similar messages Jul 09 12:02:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 12:02:23 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 12:05:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 12:05:18 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 09 12:06:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 12:06:14 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 09 12:10:00 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 09 12:10:00 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1195 previous similar messages Jul 09 12:10:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 12:10:11 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 12:12:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 12:12:32 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 09 12:15:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 12:15:35 fir-md1-s1 kernel: Lustre: Skipped 115 previous similar messages Jul 09 12:20:01 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 09 12:20:01 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1117 previous similar messages Jul 09 12:20:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 12:20:12 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 12:21:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 12:21:40 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 12:22:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 12:22:37 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 09 12:25:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 12:25:35 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 09 12:30:02 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 12:30:02 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1160 previous similar messages Jul 09 12:30:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 12:30:15 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 12:32:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 12:32:53 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 12:33:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 12:33:20 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 09 12:35:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 12:35:42 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 09 12:40:06 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 12:40:06 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1113 previous similar messages Jul 09 12:40:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 12:40:16 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 12:44:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 12:44:14 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 12:44:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 12:44:58 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 09 12:46:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 12:46:02 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 09 12:50:25 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 12:50:25 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1208 previous similar messages Jul 09 12:51:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 12:51:25 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 12:55:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 12:55:01 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 12:56:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 12:56:04 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 09 12:56:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 12:56:12 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 13:00:30 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 13:00:30 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1185 previous similar messages Jul 09 13:01:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 13:01:44 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 13:05:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 13:05:30 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 09 13:06:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 13:06:23 fir-md1-s1 kernel: Lustre: Skipped 114 previous similar messages Jul 09 13:07:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 13:07:18 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 13:10:31 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 13:10:31 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1147 previous similar messages Jul 09 13:11:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 13:11:44 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 09 13:15:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 13:15:56 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 09 13:16:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 13:16:36 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 09 13:20:34 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 13:20:34 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1186 previous similar messages Jul 09 13:21:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 13:21:48 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 13:22:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 13:22:35 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 13:26:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 13:26:00 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 09 13:26:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 13:26:41 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 09 13:29:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f27ce1a3000, cur 1562704177 expire 1562704027 last 1562703950 Jul 09 13:30:52 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 13:30:52 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1203 previous similar messages Jul 09 13:33:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 13:33:29 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 13:33:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 13:33:31 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 13:36:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f164abf1400, cur 1562704597 expire 1562704447 last 1562704370 Jul 09 13:36:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 13:36:43 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 09 13:37:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 13:37:01 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 13:40:53 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 13:40:53 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1106 previous similar messages Jul 09 13:43:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 13:43:34 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 09 13:44:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 13:44:00 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 13:46:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 13:46:46 fir-md1-s1 kernel: Lustre: Skipped 107 previous similar messages Jul 09 13:47:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 13:47:13 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 09 13:50:54 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 13:50:54 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1140 previous similar messages Jul 09 13:53:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 13:53:52 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 09 13:56:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 13:56:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 13:56:47 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 13:56:47 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 09 13:58:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 13:58:31 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 09 14:00:55 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 14:00:55 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1173 previous similar messages Jul 09 14:03:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 14:03:54 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 14:07:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 14:07:08 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 14:07:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 14:07:21 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 09 14:08:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 14:08:37 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 09 14:10:55 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 14:10:55 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1213 previous similar messages Jul 09 14:14:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 14:14:11 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 09 14:17:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 14:17:21 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 09 14:17:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25227b4c00, cur 1562707070 expire 1562706920 last 1562706843 Jul 09 14:18:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 14:18:42 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 09 14:20:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 14:20:31 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 09 14:20:58 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 09 14:20:58 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1264 previous similar messages Jul 09 14:24:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 14:24:13 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 14:27:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 14:27:31 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 09 14:29:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 14:29:05 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 14:31:02 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 14:31:02 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1353 previous similar messages Jul 09 14:31:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 14:31:51 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 14:34:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 14:34:18 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 14:37:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 14:37:56 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 09 14:40:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 14:40:18 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 14:41:05 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 14:41:05 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1368 previous similar messages Jul 09 14:42:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 14:42:00 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 14:44:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 14:44:24 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 14:48:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 14:48:08 fir-md1-s1 kernel: Lustre: Skipped 104 previous similar messages Jul 09 14:50:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 14:50:49 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 09 14:51:06 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 14:51:06 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1142 previous similar messages Jul 09 14:53:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 14:53:01 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 09 14:55:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 14:55:16 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 09 14:58:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 14:58:09 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 09 15:01:20 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 15:01:20 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1215 previous similar messages Jul 09 15:02:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 15:02:44 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 09 15:04:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 15:04:48 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 15:05:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 15:05:27 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 15:08:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 15:08:14 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 09 15:11:21 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 15:11:21 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1307 previous similar messages Jul 09 15:13:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 15:13:26 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 09 15:16:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 15:16:13 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 09 15:18:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 15:18:15 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 09 15:18:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 15:18:49 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 15:21:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ed0048c0-7f49-6510-9744-70056c2a3965 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3228e51800, cur 1562710871 expire 1562710721 last 1562710644 Jul 09 15:21:22 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 15:21:22 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1291 previous similar messages Jul 09 15:23:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 15:23:36 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 09 15:26:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 15:26:57 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 09 15:28:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 15:28:19 fir-md1-s1 kernel: Lustre: Skipped 109 previous similar messages Jul 09 15:29:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 15:29:44 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 09 15:31:26 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 15:31:26 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1297 previous similar messages Jul 09 15:34:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 15:34:10 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 09 15:36:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 15:36:59 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 15:38:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 15:38:23 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 09 15:41:28 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 15:41:28 fir-md1-s1 kernel: LustreError: 21496:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1250 previous similar messages Jul 09 15:42:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 15:42:00 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 15:45:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 15:45:05 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 09 15:47:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 15:47:00 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 09 15:48:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 15:48:26 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 09 15:51:35 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 15:51:35 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1355 previous similar messages Jul 09 15:52:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 15:55:59 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 15:55:59 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 09 15:57:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 15:57:49 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 15:58:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 15:58:42 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 09 16:01:35 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 16:01:35 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1204 previous similar messages Jul 09 16:05:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 16:05:18 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 16:06:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 16:06:32 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 09 16:07:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 16:07:49 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 09 16:08:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 16:08:47 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 09 16:11:44 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 16:11:44 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1300 previous similar messages Jul 09 16:16:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 16:16:09 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 16:17:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 16:17:03 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 16:17:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f17acad8c00, cur 1562714237 expire 1562714087 last 1562714010 Jul 09 16:17:17 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 09 16:17:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 16:17:55 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 16:18:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 16:18:48 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 09 16:21:46 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 16:21:46 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1553 previous similar messages Jul 09 16:27:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 16:27:32 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 09 16:28:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 16:28:00 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 16:28:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 16:28:12 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 09 16:29:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 16:29:16 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 09 16:32:07 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 16:32:07 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1571 previous similar messages Jul 09 16:38:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 16:38:07 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 16:39:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 09 16:39:17 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 09 16:39:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 16:39:40 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 09 16:42:14 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 16:42:14 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1216 previous similar messages Jul 09 16:43:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 16:43:47 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 16:48:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 16:48:13 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 16:49:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 16:49:18 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 09 16:50:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 16:50:26 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 09 16:52:16 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 16:52:16 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1406 previous similar messages Jul 09 16:54:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 16:54:12 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 16:58:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 16:58:28 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 16:59:41 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 16:59:41 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 09 17:00:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 17:00:26 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 09 17:02:18 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 17:02:18 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1207 previous similar messages Jul 09 17:06:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 17:06:02 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 17:08:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 17:08:44 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 17:09:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 17:09:43 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 09 17:10:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 17:10:34 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 17:12:20 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 17:12:20 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1453 previous similar messages Jul 09 17:19:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 17:19:23 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 17:19:34 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 17:19:34 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 09 17:19:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 17:19:47 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 09 17:22:24 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 17:22:24 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1219 previous similar messages Jul 09 17:22:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 17:22:42 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 09 17:29:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 17:29:41 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 09 17:29:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 17:29:51 fir-md1-s1 kernel: Lustre: Skipped 118 previous similar messages Jul 09 17:30:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 17:30:03 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 17:32:25 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 17:32:25 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1329 previous similar messages Jul 09 17:32:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 17:32:44 fir-md1-s1 kernel: Lustre: Skipped 122 previous similar messages Jul 09 17:39:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 17:39:55 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 09 17:39:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 17:39:57 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 17:40:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 17:40:52 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 17:42:29 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 17:42:29 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1126 previous similar messages Jul 09 17:44:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 17:44:53 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 17:49:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 17:49:56 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 09 17:49:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 09 17:49:57 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 17:52:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 17:52:20 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 17:52:30 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 147456 GRANT, real grant 0 Jul 09 17:52:30 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1159 previous similar messages Jul 09 17:54:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 17:54:58 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 09 17:59:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 17:59:59 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 17:59:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 17:59:59 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 09 18:02:35 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 18:02:35 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1412 previous similar messages Jul 09 18:04:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 18:04:15 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 18:05:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 18:05:20 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 18:10:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 18:10:00 fir-md1-s1 kernel: Lustre: Skipped 103 previous similar messages Jul 09 18:10:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 18:10:01 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 18:12:35 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 18:12:35 fir-md1-s1 kernel: LustreError: 46515:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1240 previous similar messages Jul 09 18:16:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 18:16:32 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 09 18:20:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 18:20:05 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 18:20:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 18:20:05 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 09 18:22:44 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 18:22:44 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1388 previous similar messages Jul 09 18:25:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 18:25:53 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 18:27:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 18:29:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 18:29:27 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 09 18:30:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 18:30:08 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 09 18:30:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 18:30:11 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 18:31:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 18:31:42 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 18:32:46 fir-md1-s1 kernel: LustreError: 21290:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 18:32:46 fir-md1-s1 kernel: LustreError: 21290:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1199 previous similar messages Jul 09 18:39:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 18:39:28 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 09 18:39:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 18:39:40 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 18:40:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 18:40:09 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 09 18:40:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 18:40:54 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 18:42:53 fir-md1-s1 kernel: LustreError: 21290:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 18:42:53 fir-md1-s1 kernel: LustreError: 21290:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1471 previous similar messages Jul 09 18:49:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 18:49:28 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 09 18:50:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 18:50:23 fir-md1-s1 kernel: Lustre: Skipped 117 previous similar messages Jul 09 18:51:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 18:51:01 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 18:52:55 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 18:52:55 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1794 previous similar messages Jul 09 18:52:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 18:52:58 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 19:00:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 19:00:12 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 09 19:00:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 19:00:23 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 09 19:01:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 19:01:07 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 09 19:02:56 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 19:02:56 fir-md1-s1 kernel: LustreError: 46585:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1154 previous similar messages Jul 09 19:04:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 19:04:09 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 19:10:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 19:10:17 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 09 19:10:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 19:10:29 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 09 19:11:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 19:11:26 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 19:12:59 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 09 19:12:59 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1240 previous similar messages Jul 09 19:16:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 19:16:49 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 19:20:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 19:20:30 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 09 19:20:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 19:20:30 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 09 19:21:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 19:21:40 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 09 19:23:07 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 19:23:07 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1383 previous similar messages Jul 09 19:25:02 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1c1f2b1000, cur 1562725502 expire 1562725352 last 1562725275 Jul 09 19:27:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 19:27:38 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 19:30:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 19:30:46 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 09 19:30:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 19:30:57 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 09 19:31:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 19:31:45 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 19:33:13 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 19:33:13 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1217 previous similar messages Jul 09 19:40:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 19:40:52 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 09 19:41:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 19:41:45 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 19:42:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 19:42:12 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 19:42:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 09 19:42:56 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 09 19:43:18 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 19:43:18 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1503 previous similar messages Jul 09 19:49:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1dc7bf7c00, cur 1562726991 expire 1562726841 last 1562726764 Jul 09 19:50:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 19:50:56 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 09 19:52:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 19:52:19 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 09 19:53:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 19:53:02 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 19:53:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 09 19:53:13 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 09 19:53:27 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 19:53:27 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1171 previous similar messages Jul 09 20:01:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 20:01:13 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 09 20:02:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 20:02:35 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 20:03:33 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 20:03:33 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1553 previous similar messages Jul 09 20:04:44 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2de89fb000, cur 1562727884 expire 1562727734 last 1562727657 Jul 09 20:05:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 20:05:14 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 20:05:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 20:05:24 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 09 20:09:42 fir-md1-s1 kernel: Lustre: 46570:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0c050bc850 x1631544738553792/t0(0) o4->d4206ce1-9dd3-fa31-a867-02061bc7b726@10.9.107.34@o2ib4:17/0 lens 2936/448 e 1 to 0 dl 1562728187 ref 2 fl Interpret:/0/0 rc 0/0 Jul 09 20:09:42 fir-md1-s1 kernel: Lustre: 46570:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 11 previous similar messages Jul 09 20:11:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 20:11:15 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 09 20:12:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 20:12:36 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 20:13:37 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 20:13:37 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1201 previous similar messages Jul 09 20:15:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 20:15:25 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 09 20:15:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 20:15:38 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 20:21:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 20:21:19 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 09 20:23:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 20:23:09 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 20:23:38 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 20:23:38 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1305 previous similar messages Jul 09 20:26:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 20:26:04 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 20:27:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 20:27:42 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 20:31:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 20:31:23 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 09 20:33:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 20:33:11 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 09 20:33:44 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 20:33:44 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1406 previous similar messages Jul 09 20:36:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 20:36:10 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 09 20:38:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 20:38:33 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 20:42:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 20:42:18 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 09 20:43:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 20:43:48 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 20:43:48 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 20:43:48 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1220 previous similar messages Jul 09 20:46:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 20:46:10 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 20:49:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 20:49:50 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 20:52:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 20:52:32 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 09 20:53:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 20:53:49 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 20:53:54 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 20:53:54 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1412 previous similar messages Jul 09 20:56:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 20:56:34 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 09 21:02:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 21:02:26 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 21:02:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 21:02:34 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 09 21:03:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 21:03:54 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 21:04:02 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 21:04:02 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1208 previous similar messages Jul 09 21:06:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 21:06:55 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 09 21:12:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 21:12:36 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 09 21:13:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 21:13:58 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 21:14:02 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 21:14:02 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1370 previous similar messages Jul 09 21:17:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 21:17:34 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 09 21:22:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 21:22:30 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 21:22:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 21:22:39 fir-md1-s1 kernel: Lustre: Skipped 107 previous similar messages Jul 09 21:24:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 21:24:01 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 21:24:07 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 21:24:07 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1263 previous similar messages Jul 09 21:27:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 21:27:55 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 09 21:32:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 21:32:43 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 09 21:34:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 21:34:05 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 09 21:34:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 21:34:15 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 09 21:34:16 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 09 21:34:16 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1296 previous similar messages Jul 09 21:38:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 21:38:06 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 09 21:42:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 21:42:47 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 09 21:44:18 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 21:44:18 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1331 previous similar messages Jul 09 21:44:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 21:44:43 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 21:45:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 21:45:23 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 21:50:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 21:50:00 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 09 21:52:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 21:52:58 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 09 21:54:20 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 21:54:20 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1662 previous similar messages Jul 09 21:54:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 21:54:49 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 21:55:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 21:55:50 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 21:55:53 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 61ae4453-9148-72f5-b1f3-a11de36a336a (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34f418a400, cur 1562734553 expire 1562734403 last 1562734326 Jul 09 21:56:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 3b5811cb-a5ba-651b-84fc-da5d7c08aeef (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f192e716c00, cur 1562734561 expire 1562734411 last 1562734334 Jul 09 22:00:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 22:00:27 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 09 22:03:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 22:03:27 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 09 22:04:22 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 22:04:22 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1382 previous similar messages Jul 09 22:05:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 22:05:14 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 09 22:06:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 22:06:00 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 09 22:10:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 22:10:35 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 09 22:13:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 22:13:29 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 09 22:14:30 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 22:14:30 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1367 previous similar messages Jul 09 22:15:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 22:15:46 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 09 22:16:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 22:16:11 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 22:21:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 22:21:25 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 09 22:23:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 22:23:31 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 09 22:24:30 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 22:24:30 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1232 previous similar messages Jul 09 22:26:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 09 22:26:09 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 22:29:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 22:29:09 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 22:31:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 22:31:32 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 09 22:33:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 09 22:33:36 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 09 22:34:35 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 22:34:35 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1220 previous similar messages Jul 09 22:36:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 22:36:24 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 22:40:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 22:40:42 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 09 22:41:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 22:41:51 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 09 22:43:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 22:43:52 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 09 22:44:35 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 22:44:35 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1450 previous similar messages Jul 09 22:46:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 09 22:46:31 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 22:51:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 22:51:22 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 22:53:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 09 22:53:56 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 09 22:54:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 22:54:10 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 09 22:54:38 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 22:54:38 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1215 previous similar messages Jul 09 22:56:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 22:56:38 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 09 23:03:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 23:03:55 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 23:04:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 23:04:13 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 09 23:04:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 23:04:23 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 09 23:04:45 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 23:04:45 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1182 previous similar messages Jul 09 23:06:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 23:06:47 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 09 23:14:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 09 23:14:17 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 09 23:14:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 23:14:24 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 09 23:14:47 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 23:14:47 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1224 previous similar messages Jul 09 23:15:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 23:15:18 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 09 23:17:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 23:17:18 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 09 23:19:54 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3f11760000, cur 1562739594 expire 1562739444 last 1562739367 Jul 09 23:19:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 09 23:24:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 23:24:20 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 09 23:24:49 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 23:24:49 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1457 previous similar messages Jul 09 23:25:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 23:25:24 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 23:26:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 23:26:44 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 09 23:27:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 23:27:20 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 09 23:34:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 23:34:22 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 09 23:34:51 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 23:34:51 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1144 previous similar messages Jul 09 23:36:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 23:36:16 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 09 23:37:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 23:37:54 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 09 23:38:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 23:38:56 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 23:44:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 23:44:24 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 09 23:44:54 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 23:44:54 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1534 previous similar messages Jul 09 23:48:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 09 23:48:02 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 09 23:48:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 09 23:48:14 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 09 23:49:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 09 23:49:05 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 09 23:54:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 09 23:54:27 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 09 23:55:02 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 09 23:55:02 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1260 previous similar messages Jul 09 23:58:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 09 23:58:06 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 09 23:59:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 09 23:59:28 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 00:03:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 00:03:07 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 00:04:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 00:04:28 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 10 00:05:15 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 00:05:15 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1344 previous similar messages Jul 10 00:08:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 00:08:14 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 10 00:09:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 00:09:34 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 10 00:14:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 00:14:02 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 00:14:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 00:14:34 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 10 00:15:16 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 00:15:16 fir-md1-s1 kernel: LustreError: 44036:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1363 previous similar messages Jul 10 00:18:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 00:18:51 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 10 00:21:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 10 00:21:36 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 00:24:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 00:24:35 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 10 00:25:28 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 00:25:28 fir-md1-s1 kernel: LustreError: 21454:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1342 previous similar messages Jul 10 00:26:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 00:26:02 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 00:29:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 00:29:01 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 10 00:32:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 00:32:46 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 10 00:34:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 00:34:35 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 10 00:35:28 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 00:35:28 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1376 previous similar messages Jul 10 00:36:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 00:36:11 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 10 00:39:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 00:39:27 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 00:43:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 00:43:08 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 10 00:44:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 00:44:49 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 10 00:45:36 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 00:45:36 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1223 previous similar messages Jul 10 00:49:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 00:49:37 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 00:53:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 00:53:28 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 10 00:53:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f8ad0c800, cur 1562745223 expire 1562745073 last 1562744996 Jul 10 00:54:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 00:54:53 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 10 00:55:40 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 00:55:40 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1270 previous similar messages Jul 10 00:59:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 00:59:48 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 10 01:04:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 01:04:25 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 01:04:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 01:04:54 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 10 01:05:54 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 10 01:05:54 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1264 previous similar messages Jul 10 01:10:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 01:10:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 01:10:19 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 01:12:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 01:12:19 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 01:14:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 01:14:31 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 10 01:14:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 01:14:59 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 10 01:15:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 01:16:04 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 01:16:04 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1378 previous similar messages Jul 10 01:20:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 01:20:19 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 01:24:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 01:24:30 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 01:24:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 01:24:37 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 10 01:25:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 01:25:03 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 10 01:26:05 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 01:26:05 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1282 previous similar messages Jul 10 01:31:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 01:31:09 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 10 01:34:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 01:34:36 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 01:35:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 01:35:03 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 10 01:35:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 01:35:03 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 10 01:36:08 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 10 01:36:08 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1274 previous similar messages Jul 10 01:41:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 01:41:25 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 10 01:44:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 01:44:42 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 01:45:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 01:45:04 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 10 01:46:10 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 01:46:10 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1440 previous similar messages Jul 10 01:46:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 01:46:19 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 10 01:51:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 01:51:50 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 10 01:54:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 01:54:51 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 10 01:55:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 01:55:08 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 10 01:56:15 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 01:56:15 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1302 previous similar messages Jul 10 01:56:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 01:56:24 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 10 02:01:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 02:01:56 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 02:06:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 02:06:00 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 10 02:06:23 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 02:06:23 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1413 previous similar messages Jul 10 02:06:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 02:06:29 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 02:09:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 02:09:30 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 02:12:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 02:12:19 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 02:16:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 02:16:28 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 10 02:16:35 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 32768 GRANT, real grant 0 Jul 10 02:16:35 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 910 previous similar messages Jul 10 02:17:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 02:17:51 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 10 02:22:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 02:22:33 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 02:23:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 02:23:44 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 02:26:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 02:26:33 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 10 02:26:36 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 02:26:36 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 746 previous similar messages Jul 10 02:27:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 02:27:55 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 10 02:32:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 02:32:41 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 02:36:37 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 10 02:36:37 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 624 previous similar messages Jul 10 02:36:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 02:36:48 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 10 02:39:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 10 02:39:42 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 10 02:42:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 02:42:56 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 02:43:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24208bec00, cur 1562751820 expire 1562751670 last 1562751593 Jul 10 02:46:40 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 10 02:46:40 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 721 previous similar messages Jul 10 02:46:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2ccef14c00, cur 1562752003 expire 1562751853 last 1562751776 Jul 10 02:46:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 02:46:58 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 02:50:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 02:50:41 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 10 02:52:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 02:52:05 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 10 02:52:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 02:52:54 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 02:53:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 02:53:35 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 10 02:56:41 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 02:56:41 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 776 previous similar messages Jul 10 02:57:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 02:57:24 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 02:57:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 03:03:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 03:03:00 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 10 03:03:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 03:03:53 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 03:06:44 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 03:06:44 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 824 previous similar messages Jul 10 03:07:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 03:07:00 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 03:07:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 03:07:51 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 10 03:13:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 03:13:07 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 10 03:14:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 03:14:05 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 03:16:45 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 03:16:45 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 554 previous similar messages Jul 10 03:17:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 03:17:53 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 10 03:21:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 03:21:08 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 03:23:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 03:23:24 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 10 03:24:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 03:24:08 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 03:25:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client f340038f-9ca0-b54d-f024-5ea93ca12997 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f41e39000, cur 1562754351 expire 1562754201 last 1562754124 Jul 10 03:26:48 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 10 03:26:48 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 303 previous similar messages Jul 10 03:28:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 03:28:07 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 10 03:33:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 10 03:33:40 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 10 03:34:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 03:34:32 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 03:37:06 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 10 03:37:06 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 327 previous similar messages Jul 10 03:38:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 03:38:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 03:38:08 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 10 03:38:08 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 03:43:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 03:43:42 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 10 03:44:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 03:44:37 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 10 03:47:09 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 03:47:09 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 506 previous similar messages Jul 10 03:48:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 03:48:14 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 10 03:50:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 03:50:05 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 03:55:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 03:55:19 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 10 03:55:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 10 03:55:23 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 10 03:57:18 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 03:57:18 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 615 previous similar messages Jul 10 03:58:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 03:58:14 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 10 04:05:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 04:05:28 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 04:05:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 04:05:36 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 04:06:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 04:06:09 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 04:07:22 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 04:07:22 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 532 previous similar messages Jul 10 04:08:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 04:08:14 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 10 04:15:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 04:15:41 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 10 04:16:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 04:16:34 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 04:17:24 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 04:17:24 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 675 previous similar messages Jul 10 04:18:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 04:18:20 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 10 04:25:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 04:25:13 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 04:25:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 04:25:41 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 10 04:27:26 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 04:27:26 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 715 previous similar messages Jul 10 04:28:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 04:28:09 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 10 04:28:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 04:28:21 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 10 04:35:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 04:35:29 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 04:36:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 04:36:20 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 10 04:37:28 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 04:37:28 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 746 previous similar messages Jul 10 04:38:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 04:38:10 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 10 04:38:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 04:38:22 fir-md1-s1 kernel: Lustre: Skipped 111 previous similar messages Jul 10 04:39:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 81ca2f92-1c99-e8fa-d30d-6f44b638b624 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2f16a61000, cur 1562758780 expire 1562758630 last 1562758553 Jul 10 04:39:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 04:46:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 04:46:48 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 04:47:29 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 04:47:29 fir-md1-s1 kernel: LustreError: 57558:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 538 previous similar messages Jul 10 04:48:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 04:48:22 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 10 04:49:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 04:49:14 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 10 04:57:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 04:57:01 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 04:57:29 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 04:57:29 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 468 previous similar messages Jul 10 04:58:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 04:58:23 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 10 04:58:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 04:58:59 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 04:59:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 04:59:19 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 10 05:00:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 05:00:33 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 05:04:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 05:04:52 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 05:07:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 05:07:04 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 10 05:07:53 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 05:07:53 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 288 previous similar messages Jul 10 05:08:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 05:08:28 fir-md1-s1 kernel: Lustre: Skipped 124 previous similar messages Jul 10 05:09:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 10 05:09:26 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 10 05:17:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 05:17:13 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 05:17:56 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 05:17:56 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 480 previous similar messages Jul 10 05:18:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 05:18:29 fir-md1-s1 kernel: Lustre: Skipped 109 previous similar messages Jul 10 05:19:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 05:19:27 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 10 05:20:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 05:20:06 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 05:25:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 05:27:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 05:27:14 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 05:27:56 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 05:27:56 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 476 previous similar messages Jul 10 05:28:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 05:28:36 fir-md1-s1 kernel: Lustre: Skipped 108 previous similar messages Jul 10 05:29:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 05:29:23 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 05:29:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 05:29:29 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 10 05:34:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 05:37:58 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 05:37:58 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 658 previous similar messages Jul 10 05:38:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 05:38:00 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 05:38:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 05:38:42 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 10 05:40:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 05:40:03 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 05:42:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 05:47:59 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 05:47:59 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 568 previous similar messages Jul 10 05:48:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 05:48:00 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 05:48:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 05:48:57 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 10 05:50:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 05:50:12 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 10 05:56:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 05:56:54 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 05:58:01 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 05:58:01 fir-md1-s1 kernel: LustreError: 20500:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 696 previous similar messages Jul 10 05:58:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 05:58:14 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 05:59:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 05:59:03 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Jul 10 06:00:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 06:00:23 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 10 06:08:06 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 06:08:06 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 759 previous similar messages Jul 10 06:08:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 06:08:17 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 06:09:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 06:09:27 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 10 06:10:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 06:10:23 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 06:14:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 06:14:32 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 06:18:08 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 06:18:08 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 580 previous similar messages Jul 10 06:18:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 06:18:34 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 10 06:19:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 06:19:39 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 06:20:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 06:20:37 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 10 06:28:16 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 06:28:16 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 528 previous similar messages Jul 10 06:28:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 06:28:43 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 06:29:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 06:29:51 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 10 06:30:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 06:30:26 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 06:32:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 06:32:39 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 10 06:38:18 fir-md1-s1 kernel: Lustre: 23597:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2f100ed100 x1638075991861936/t0(0) o101->b041cef5-fff9-4fc6-cc5f-62c5a80e124b@10.9.0.81@o2ib4:23/0 lens 480/568 e 1 to 0 dl 1562765903 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 06:38:25 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 06:38:25 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 548 previous similar messages Jul 10 06:38:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b041cef5-fff9-4fc6-cc5f-62c5a80e124b (at 10.9.0.81@o2ib4) reconnecting Jul 10 06:38:45 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 06:39:09 fir-md1-s1 kernel: Lustre: 22007:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1ad960b600 x1638276144880736/t0(0) o36->ef0748a0-58bc-3624-ed96-74860cd1e591@10.8.0.66@o2ib6:14/0 lens 512/2888 e 0 to 0 dl 1562765954 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 06:39:33 fir-md1-s1 kernel: LustreError: 23704:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562765883, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f212eaf5a00/0x5d9ee638bed62f8c lrc: 3/0,1 mode: --/PW res: [0x200029cf7:0x9d:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23704 timeout: 0 lvb_type: 0 Jul 10 06:39:40 fir-md1-s1 kernel: Lustre: 10582:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-19), not sending early reply req@ffff8f2ecce84800 x1638276144883648/t0(0) o101->ef0748a0-58bc-3624-ed96-74860cd1e591@10.8.0.66@o2ib6:15/0 lens 576/3264 e 0 to 0 dl 1562765985 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 06:40:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 06:40:08 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 10 06:40:14 fir-md1-s1 kernel: LustreError: 22283:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562765924, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f24f26e8480/0x5d9ee638bef3c93d lrc: 3/0,1 mode: --/EX res: [0x200029cf7:0x9d:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22283 timeout: 0 lvb_type: 0 Jul 10 06:40:31 fir-md1-s1 kernel: LustreError: 50582:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562765941, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f0ab91d5e80/0x5d9ee638befea93e lrc: 3/1,0 mode: --/PR res: [0x200025ce2:0x1fa1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 50582 timeout: 0 lvb_type: 0 Jul 10 06:40:32 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.9.0.81@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff8f22955086c0/0x5d9ee638bed60c69 lrc: 3/0,0 mode: PR/PR res: [0x200029cf7:0x9d:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.9.0.81@o2ib4 remote: 0x483a08d1111db65 expref: 46 pid: 21430 timeout: 1881092 lvb_type: 0 Jul 10 06:40:32 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Jul 10 06:40:32 fir-md1-s1 kernel: LustreError: 23704:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f20269e5800 ns: mdt-fir-MDT0000_UUID lock: ffff8f212eaf5a00/0x5d9ee638bed62f8c lrc: 3/0,0 mode: PW/PW res: [0x200029cf7:0x9d:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT flags: 0x50200000000000 nid: 10.9.0.81@o2ib4 remote: 0x483a08d1111db73 expref: 33 pid: 23704 timeout: 0 lvb_type: 0 Jul 10 06:44:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client b041cef5-fff9-4fc6-cc5f-62c5a80e124b (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f37b3b6c800, cur 1562766259 expire 1562766109 last 1562766032 Jul 10 06:44:19 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 06:44:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 10 06:44:48 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 06:48:34 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 10 06:48:34 fir-md1-s1 kernel: LustreError: 21996:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 448 previous similar messages Jul 10 06:48:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 06:48:48 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 06:50:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 06:50:21 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 10 06:55:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 06:55:26 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 10 06:58:36 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 06:58:36 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 627 previous similar messages Jul 10 06:59:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 06:59:00 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 07:00:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 07:00:28 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 10 07:03:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 07:03:33 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 07:04:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 07:05:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 07:05:55 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 07:08:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 07:08:43 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 07:08:43 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 596 previous similar messages Jul 10 07:09:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 07:09:12 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 07:10:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 07:10:29 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 07:17:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 07:18:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 07:18:25 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 10 07:18:45 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 07:18:45 fir-md1-s1 kernel: LustreError: 46577:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 607 previous similar messages Jul 10 07:19:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 07:19:25 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 07:20:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 07:20:44 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 10 07:27:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 07:27:41 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 07:28:47 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 07:28:47 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 594 previous similar messages Jul 10 07:29:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 07:29:43 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jul 10 07:30:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 07:30:01 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 10 07:30:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 07:30:51 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 07:38:49 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 07:38:49 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 700 previous similar messages Jul 10 07:39:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 07:39:44 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 10 07:40:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 07:40:10 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 10 07:41:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 07:41:01 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 10 07:42:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 07:42:42 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 10 07:48:55 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 07:48:55 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 588 previous similar messages Jul 10 07:50:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 07:50:11 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 07:51:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 07:51:16 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 10 07:52:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 07:52:36 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 10 07:55:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 07:55:00 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 07:58:56 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 07:58:56 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 486 previous similar messages Jul 10 07:59:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 2bad2d99-434c-c071-8f86-46075da8e78f (at 10.9.115.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0af8d72800, cur 1562770761 expire 1562770611 last 1562770534 Jul 10 08:01:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 08:01:12 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 08:01:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 08:01:29 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 08:03:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 08:03:08 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 10 08:05:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 08:05:27 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 08:09:07 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 08:09:07 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 464 previous similar messages Jul 10 08:12:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 08:12:18 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 08:12:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 08:12:18 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 08:13:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 08:13:33 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 10 08:15:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 08:15:52 fir-md1-s1 kernel: LustreError: Skipped 13 previous similar messages Jul 10 08:19:09 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 32768 GRANT, real grant 0 Jul 10 08:19:09 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 436 previous similar messages Jul 10 08:22:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 08:22:35 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 08:22:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 08:22:35 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 10 08:23:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 08:23:41 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jul 10 08:26:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 08:26:42 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 10 08:29:16 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 10 08:29:16 fir-md1-s1 kernel: LustreError: 21995:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 558 previous similar messages Jul 10 08:32:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 08:32:50 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 08:32:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 08:32:50 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 10 08:36:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 08:36:28 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 10 08:36:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 08:36:49 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 10 08:39:23 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 08:39:23 fir-md1-s1 kernel: LustreError: 27580:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 604 previous similar messages Jul 10 08:42:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f20a9c2a400, cur 1562773351 expire 1562773201 last 1562773124 Jul 10 08:42:31 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 08:43:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 08:43:20 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 10 08:43:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 08:43:35 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 08:47:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 08:47:00 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 10 08:48:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 08:48:45 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 10 08:49:25 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 08:49:25 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 588 previous similar messages Jul 10 08:53:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 08:53:26 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 10 08:53:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 08:53:41 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 08:57:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 08:57:02 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 10 08:59:27 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 08:59:27 fir-md1-s1 kernel: LustreError: 21385:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 502 previous similar messages Jul 10 09:00:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 09:00:51 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 10 09:02:33 fir-md1-s1 kernel: Lustre: 21452:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2ef25f9e00 x1638088148657920/t0(0) o101->5282ca62-d94c-33fd-9d61-31ebcd98e0af@10.9.116.1@o2ib4:8/0 lens 576/3264 e 1 to 0 dl 1562774558 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 09:02:33 fir-md1-s1 kernel: Lustre: 21452:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 282 previous similar messages Jul 10 09:02:34 fir-md1-s1 kernel: Lustre: 21333:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2ecb5a1b00 x1634138221307008/t0(0) o101->cfa699f5-5c9c-ea69-d701-26f52d68dba1@10.9.101.37@o2ib4:9/0 lens 328/0 e 1 to 0 dl 1562774559 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 10 09:02:34 fir-md1-s1 kernel: Lustre: 21333:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 288 previous similar messages Jul 10 09:02:35 fir-md1-s1 kernel: Lustre: 22284:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1ec5385a00 x1636456297044992/t0(0) o101->05e7d18b-fd1f-bd0e-dca1-20091393d8f8@10.9.108.66@o2ib4:10/0 lens 576/0 e 1 to 0 dl 1562774560 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 10 09:02:35 fir-md1-s1 kernel: Lustre: 22284:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 139 previous similar messages Jul 10 09:02:37 fir-md1-s1 kernel: Lustre: 21378:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f38b7611200 x1636457995286064/t0(0) o101->da0ec9bf-1999-ba8d-5389-20d1ebbaa0f5@10.9.107.72@o2ib4:12/0 lens 576/3264 e 1 to 0 dl 1562774562 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 09:02:37 fir-md1-s1 kernel: Lustre: 21378:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 217 previous similar messages Jul 10 09:02:41 fir-md1-s1 kernel: Lustre: 21678:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f38b7612100 x1631604556695008/t0(0) o101->7f8dc145-a081-da87-1da4-154358301486@10.9.108.1@o2ib4:16/0 lens 576/3264 e 1 to 0 dl 1562774566 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 09:02:41 fir-md1-s1 kernel: Lustre: 21678:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1108 previous similar messages Jul 10 09:02:47 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.9.0.81@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f3692f3b840/0x5d9ee638b6b6626a lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 886 type: IBT flags: 0x60200400000020 nid: 10.9.0.81@o2ib4 remote: 0x483a08d1110c556 expref: 18 pid: 20554 timeout: 1889627 lvb_type: 0 Jul 10 09:02:48 fir-md1-s1 kernel: Lustre: 23704:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:9s); client may timeout. req@ffff8f26a7be8000 x1631558636133808/t0(0) o101->9c58438d-335a-1a4a-8b6e-0ac0b859df8d@10.8.12.23@o2ib6:8/0 lens 576/0 e 1 to 0 dl 1562774558 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 10 09:02:48 fir-md1-s1 kernel: Lustre: 23684:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:9s); client may timeout. req@ffff8f348ab51200 x1631558636133888/t0(0) o101->9c58438d-335a-1a4a-8b6e-0ac0b859df8d@10.8.12.23@o2ib6:8/0 lens 576/0 e 1 to 0 dl 1562774558 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 10 09:02:48 fir-md1-s1 kernel: LustreError: 20545:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.10.21@o2ib6: deadline 20:4s ago req@ffff8f21dc32f200 x1632260917013200/t0(0) o101->64093eed-1899-7457-95e6-ff7526581ffb@10.8.10.21@o2ib6:13/0 lens 576/0 e 0 to 0 dl 1562774563 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jul 10 09:02:48 fir-md1-s1 kernel: LustreError: 26254:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.10.21@o2ib6: deadline 20:4s ago req@ffff8f25e2c06000 x1632260917013168/t0(0) o101->64093eed-1899-7457-95e6-ff7526581ffb@10.8.10.21@o2ib6:13/0 lens 576/0 e 0 to 0 dl 1562774563 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jul 10 09:02:48 fir-md1-s1 kernel: LustreError: 20545:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 40 previous similar messages Jul 10 09:02:48 fir-md1-s1 kernel: LustreError: 26254:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 40 previous similar messages Jul 10 09:02:48 fir-md1-s1 kernel: Lustre: 23704:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1501 previous similar messages Jul 10 09:03:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 09:03:35 fir-md1-s1 kernel: Lustre: Skipped 669 previous similar messages Jul 10 09:03:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 09:03:47 fir-md1-s1 kernel: Lustre: Skipped 657 previous similar messages Jul 10 09:06:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client b041cef5-fff9-4fc6-cc5f-62c5a80e124b (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f442670e400, cur 1562774810 expire 1562774660 last 1562774583 Jul 10 09:09:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 09:09:20 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 10 09:09:32 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 09:09:32 fir-md1-s1 kernel: LustreError: 21388:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 627 previous similar messages Jul 10 09:10:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 09:10:55 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 09:13:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 09:13:43 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 10 09:14:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 09:14:08 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 09:19:34 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 10 09:19:34 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 523 previous similar messages Jul 10 09:20:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 09:20:45 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 10 09:21:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 09:21:55 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 10 09:23:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 09:23:43 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 09:24:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 09:24:39 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 09:25:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 96bed43a-b7c9-0e49-67fd-9247dc304082 (at 10.8.30.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2508cc6c00, cur 1562775927 expire 1562775777 last 1562775700 Jul 10 09:29:41 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 09:29:41 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 597 previous similar messages Jul 10 09:32:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 10 09:32:32 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 10 09:33:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 09:33:53 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 10 09:34:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 09:34:39 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 10 09:35:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 09:35:05 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 10 09:39:43 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 09:39:43 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 467 previous similar messages Jul 10 09:43:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 09:43:36 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 10 09:44:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 09:44:06 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 09:45:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f23ea245400, cur 1562777120 expire 1562776970 last 1562776893 Jul 10 09:45:20 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 10 09:45:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 09:45:25 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 10 09:45:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 09:45:51 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 09:47:25 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f16618f8c00, cur 1562777245 expire 1562777095 last 1562777018 Jul 10 09:49:45 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 09:49:45 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 565 previous similar messages Jul 10 09:53:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 10 09:53:56 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 10 09:54:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 48055890-ac7a-40c2-f14b-00e7fd6a0cc0 (at 10.8.22.30@o2ib6) Jul 10 09:54:07 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 10 09:55:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d75f4c800, cur 1562777749 expire 1562777599 last 1562777522 Jul 10 09:56:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 09:56:14 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 09:57:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 09:57:04 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 10 09:57:07 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2d8d221000, cur 1562777827 expire 1562777677 last 1562777600 Jul 10 09:59:45 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 09:59:45 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 570 previous similar messages Jul 10 10:04:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 10:04:33 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 10 10:05:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 10 10:05:26 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 10 10:07:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 10:07:18 fir-md1-s1 kernel: LustreError: Skipped 14 previous similar messages Jul 10 10:07:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 10:07:20 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 10 10:09:47 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 10 10:09:47 fir-md1-s1 kernel: LustreError: 46513:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 714 previous similar messages Jul 10 10:14:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 10:14:46 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 10 10:17:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 10:17:08 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 10 10:17:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 10:17:27 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 10:18:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 10:18:57 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 10:19:50 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 10:19:50 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 708 previous similar messages Jul 10 10:19:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f22a1520c00, cur 1562779193 expire 1562779043 last 1562778966 Jul 10 10:24:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 10:24:51 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 10 10:27:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 10:27:29 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 10 10:27:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 10:27:30 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 10:29:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 10:29:21 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 10 10:29:57 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 10:29:57 fir-md1-s1 kernel: LustreError: 46565:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 711 previous similar messages Jul 10 10:35:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 10:35:17 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 10:37:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 10:37:30 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 10:37:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 10:37:55 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 10 10:39:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 10:39:29 fir-md1-s1 kernel: LustreError: Skipped 16 previous similar messages Jul 10 10:40:09 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 10:40:09 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 656 previous similar messages Jul 10 10:45:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 10:45:20 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 10 10:47:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 10:47:48 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 10 10:48:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 10:48:44 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 10 10:50:11 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 10:50:11 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 651 previous similar messages Jul 10 10:52:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 10:52:21 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 10:55:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 10:55:21 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 10 10:58:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 10:58:01 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 10:58:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 10:58:49 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 10 11:00:20 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 10 11:00:20 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 713 previous similar messages Jul 10 11:02:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 11:02:52 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 10 11:05:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 11:05:36 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 11:08:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 11:08:59 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 10 11:09:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 11:09:45 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 10 11:10:20 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 11:10:20 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 583 previous similar messages Jul 10 11:14:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 11:14:10 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 10 11:15:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 11:15:56 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 11:19:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 11:19:17 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 11:20:20 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 11:20:20 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 624 previous similar messages Jul 10 11:20:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 11:20:49 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 10 11:26:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 11:26:15 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 10 11:28:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 11:28:02 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 11:29:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 11:29:23 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 10 11:30:21 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 10 11:30:21 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 548 previous similar messages Jul 10 11:31:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 11:31:07 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 10 11:36:14 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2efcb44000, cur 1562783774 expire 1562783624 last 1562783547 Jul 10 11:36:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 11:36:16 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 11:38:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 11:38:28 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 11:39:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 11:39:31 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 10 11:40:24 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 11:40:24 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 608 previous similar messages Jul 10 11:41:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 11:41:39 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jul 10 11:44:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f30f07c00, cur 1562784274 expire 1562784124 last 1562784047 Jul 10 11:46:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 11:46:35 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 10 11:48:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 11:48:52 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 11:49:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 11:49:33 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 11:50:29 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 11:50:29 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 627 previous similar messages Jul 10 11:51:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 10 11:51:45 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 10 11:53:41 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client e912b1f0-4b75-614b-7c78-541b22033095 (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2fb52cac00, cur 1562784821 expire 1562784671 last 1562784594 Jul 10 11:56:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 11:56:36 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 10 11:59:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 11:59:34 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 11:59:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 11:59:55 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 10 12:00:31 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 12:00:31 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 737 previous similar messages Jul 10 12:02:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 10 12:02:49 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 10 12:06:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 12:06:49 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 10 12:09:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 55cf06b7-ada2-2c2a-4329-eb93e8b4cb23 (at 10.9.104.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f19630e0000, cur 1562785766 expire 1562785616 last 1562785539 Jul 10 12:09:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 55cf06b7-ada2-2c2a-4329-eb93e8b4cb23 (at 10.9.104.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34e17d3400, cur 1562785775 expire 1562785625 last 1562785548 Jul 10 12:09:35 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 12:10:34 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 12:10:34 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 751 previous similar messages Jul 10 12:10:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 12:10:48 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 12:12:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 12:12:23 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 12:13:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 12:13:00 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 10 12:17:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 12:17:03 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 12:20:40 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 10 12:20:40 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 693 previous similar messages Jul 10 12:20:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 12:20:49 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 10 12:22:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 12:22:39 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 12:23:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 12:23:16 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 10 12:27:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 12:27:39 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 10 12:30:42 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 12:30:42 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 673 previous similar messages Jul 10 12:30:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 12:30:50 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 10 12:33:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 12:33:46 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 12:34:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 12:34:05 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 10 12:37:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 12:37:46 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 10 12:40:48 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 10 12:40:48 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 624 previous similar messages Jul 10 12:40:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 12:40:58 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 12:44:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 12:44:42 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 12:45:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 12:45:10 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 12:47:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 12:47:51 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 10 12:50:49 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 10 12:50:49 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 609 previous similar messages Jul 10 12:52:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 12:52:51 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 10 12:55:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 12:55:19 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 12:57:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 12:57:53 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 12:58:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 12:58:42 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 10 13:00:53 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 13:00:53 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 544 previous similar messages Jul 10 13:02:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 13:02:57 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 13:05:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 13:05:29 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 13:07:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 13:07:55 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 10 13:08:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 13:08:49 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 13:10:55 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 13:10:55 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 619 previous similar messages Jul 10 13:13:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 13:13:05 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 13:15:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 13:15:30 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 10 13:18:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 13:18:18 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 10 13:18:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 13:18:57 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 10 13:21:01 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 10 13:21:01 fir-md1-s1 kernel: LustreError: 21987:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 601 previous similar messages Jul 10 13:23:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 13:23:12 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 13:26:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 13:26:00 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 10 13:28:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 13:28:51 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 10 13:29:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 13:29:56 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 13:31:06 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 13:31:06 fir-md1-s1 kernel: LustreError: 66901:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 761 previous similar messages Jul 10 13:33:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 13:33:23 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 10 13:36:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 13:36:02 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 10 13:38:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 13:38:53 fir-md1-s1 kernel: Lustre: Skipped 112 previous similar messages Jul 10 13:41:14 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 13:41:14 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 631 previous similar messages Jul 10 13:43:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 13:43:40 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 13:44:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 13:44:38 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 10 13:46:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 13:46:12 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 10 13:49:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 13:49:03 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 10 13:51:15 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 13:51:15 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 790 previous similar messages Jul 10 13:53:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ae14ed36-ba60-4740-8815-84f6adaeeb15 (at 10.9.114.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f453e965800, cur 1562791993 expire 1562791843 last 1562791766 Jul 10 13:53:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 13:53:47 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 10 13:56:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 13:56:18 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 13:56:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 13:56:21 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 10 13:59:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 13:59:06 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 10 14:01:15 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 14:01:15 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 737 previous similar messages Jul 10 14:03:43 fir-md1-s1 kernel: Lustre: 23758:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562792616/real 1562792616] req@ffff8f2644ac4b00 x1636729522294720/t0(0) o104->fir-MDT0002@10.8.17.20@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562792623 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 10 14:03:50 fir-md1-s1 kernel: Lustre: 23758:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562792623/real 1562792623] req@ffff8f2644ac4b00 x1636729522294720/t0(0) o104->fir-MDT0002@10.8.17.20@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562792630 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 14:03:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6ad7e9e1-dbc5-f9a1-bdd9-743173a51d0b (at 10.8.17.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f17f9783400, cur 1562792631 expire 1562792481 last 1562792404 Jul 10 14:03:51 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 14:03:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 14:03:51 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 10 14:06:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 14:06:21 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 14:06:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 14:06:22 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 10 14:09:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 14:09:49 fir-md1-s1 kernel: Lustre: Skipped 97 previous similar messages Jul 10 14:11:23 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 14:11:23 fir-md1-s1 kernel: LustreError: 22269:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 573 previous similar messages Jul 10 14:14:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 14:14:12 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 10 14:16:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 14:16:22 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 10 14:18:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 14:18:21 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 10 14:19:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 14:19:51 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 10 14:21:24 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 14:21:24 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 623 previous similar messages Jul 10 14:24:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 14:24:28 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 14:27:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 14:27:05 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 10 14:28:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 14:28:36 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 14:30:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 14:30:38 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 10 14:31:31 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 14:31:31 fir-md1-s1 kernel: LustreError: 20501:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 614 previous similar messages Jul 10 14:34:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 14:34:35 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 14:37:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 14:37:48 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 14:40:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 14:40:52 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 10 14:41:32 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 14:41:32 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 665 previous similar messages Jul 10 14:42:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 14:42:30 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 14:44:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 14:44:43 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 14:47:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 14:47:54 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 14:51:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 14:51:02 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 10 14:51:34 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 14:51:34 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 593 previous similar messages Jul 10 14:54:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 14:54:54 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 10 14:57:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 14:57:08 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 14:59:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 14:59:21 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 10 15:01:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 15:01:08 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 10 15:01:36 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 15:01:36 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 687 previous similar messages Jul 10 15:05:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 15:05:00 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 10 15:09:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 15:09:23 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 10 15:11:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 15:11:09 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 10 15:11:37 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 15:11:37 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 658 previous similar messages Jul 10 15:14:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 15:14:45 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 15:14:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e097a71c-e88a-824e-4bcb-410d766486a5 (at 10.9.114.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fbf52400, cur 1562796886 expire 1562796736 last 1562796659 Jul 10 15:14:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 15:15:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 15:15:24 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 15:16:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c50a2569-5f68-c0c4-a8b8-bfb61fe4dbbb (at 10.9.114.5@o2ib4) in 215 seconds. I think it's dead, and I am evicting it. exp ffff8f453868f000, cur 1562796962 expire 1562796812 last 1562796747 Jul 10 15:16:02 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 15:19:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 15:19:23 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 10 15:21:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 15:21:10 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 10 15:21:43 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 15:21:43 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 729 previous similar messages Jul 10 15:25:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 15:25:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 15:25:29 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 15:29:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 15:29:32 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 10 15:31:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 15:31:11 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 10 15:31:46 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 15:31:46 fir-md1-s1 kernel: LustreError: 46567:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 648 previous similar messages Jul 10 15:35:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 15:35:29 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 15:36:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 15:39:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 15:39:42 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 10 15:40:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c33dfd3e-93e2-b1e4-c92b-6be01740e2e1 (at 10.9.115.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fdbd6000, cur 1562798440 expire 1562798290 last 1562798213 Jul 10 15:40:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 15:41:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 15:41:28 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 10 15:41:48 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 15:41:48 fir-md1-s1 kernel: LustreError: 46570:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 650 previous similar messages Jul 10 15:45:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 15:45:36 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 10 15:49:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 15:49:45 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 15:51:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 15:51:34 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 10 15:51:50 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 15:51:50 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 608 previous similar messages Jul 10 15:53:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f250d936800, cur 1562799220 expire 1562799070 last 1562798993 Jul 10 15:53:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 15:55:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 15:55:46 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 10 15:59:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 15:59:30 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 15:59:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 15:59:58 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 16:01:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 16:01:35 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 10 16:01:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 16:01:41 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 16:01:51 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 16:01:51 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 678 previous similar messages Jul 10 16:05:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 16:05:57 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 16:07:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0b6cd9d400, cur 1562800054 expire 1562799904 last 1562799827 Jul 10 16:10:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 16:10:00 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 16:10:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 16:10:08 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 10 16:11:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 16:11:36 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 10 16:11:54 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 16:11:54 fir-md1-s1 kernel: LustreError: 46537:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 689 previous similar messages Jul 10 16:16:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 16:16:05 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 16:20:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 16:20:09 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 10 16:21:57 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 16:21:57 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 724 previous similar messages Jul 10 16:21:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 16:21:58 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 10 16:22:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 16:26:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 16:26:32 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 10 16:30:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 16:30:49 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 10 16:31:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 16:31:59 fir-md1-s1 kernel: Lustre: Skipped 113 previous similar messages Jul 10 16:32:04 fir-md1-s1 kernel: LustreError: 22974:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 16:32:04 fir-md1-s1 kernel: LustreError: 22974:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 633 previous similar messages Jul 10 16:35:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 16:35:14 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 16:37:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 16:37:10 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 16:40:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 16:40:52 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 10 16:42:05 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 16:42:05 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 708 previous similar messages Jul 10 16:42:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 16:42:07 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 10 16:45:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 16:45:32 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 16:47:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 16:47:11 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 10 16:51:04 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562802657/real 1562802657] req@ffff8f1e43d69500 x1636729669870912/t0(0) o104->fir-MDT0002@10.9.106.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562802664 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 10 16:51:11 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562802664/real 1562802664] req@ffff8f1e43d69500 x1636729669870912/t0(0) o104->fir-MDT0002@10.9.106.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562802671 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 16:51:12 fir-md1-s1 kernel: Lustre: 97661:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2126bc6600 x1631551896846384/t0(0) o36->6b95ce2a-f8e3-6a6f-1394-30bdffccf512@10.9.105.51@o2ib4:17/0 lens 504/2888 e 1 to 0 dl 1562802677 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 16:51:12 fir-md1-s1 kernel: Lustre: 97661:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 579 previous similar messages Jul 10 16:51:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 16:51:18 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 16:51:18 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562802671/real 1562802671] req@ffff8f1e43d69500 x1636729669870912/t0(0) o104->fir-MDT0002@10.9.106.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562802678 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 16:51:25 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562802678/real 1562802678] req@ffff8f1e43d69500 x1636729669870912/t0(0) o104->fir-MDT0002@10.9.106.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562802685 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 16:51:32 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562802685/real 1562802685] req@ffff8f1e43d69500 x1636729669870912/t0(0) o104->fir-MDT0002@10.9.106.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562802692 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 16:51:46 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562802699/real 1562802699] req@ffff8f1e43d69500 x1636729669870912/t0(0) o104->fir-MDT0002@10.9.106.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562802706 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 16:51:46 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 10 16:52:07 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562802720/real 1562802720] req@ffff8f1e43d69500 x1636729669870912/t0(0) o104->fir-MDT0002@10.9.106.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562802727 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 16:52:07 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 10 16:52:08 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 16:52:08 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 687 previous similar messages Jul 10 16:52:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 16:52:10 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 10 16:52:42 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562802755/real 1562802755] req@ffff8f1e43d69500 x1636729669870912/t0(0) o104->fir-MDT0002@10.9.106.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562802762 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 16:52:42 fir-md1-s1 kernel: Lustre: 21446:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 10 16:53:31 fir-md1-s1 kernel: LustreError: 21446:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.106.8@o2ib4) failed to reply to blocking AST (req@ffff8f1e43d69500 x1636729669870912 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f2f73dc0d80/0x5d9ee639672d158c lrc: 4/0,0 mode: PR/PR res: [0x2c002c429:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.9.106.8@o2ib4 remote: 0x9d2381126b1c50d2 expref: 387 pid: 21677 timeout: 1918013 lvb_type: 0 Jul 10 16:53:31 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.9.106.8@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 10 16:53:31 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.9.106.8@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f2f73dc0d80/0x5d9ee639672d158c lrc: 3/0,0 mode: PR/PR res: [0x2c002c429:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.9.106.8@o2ib4 remote: 0x9d2381126b1c50d2 expref: 388 pid: 21677 timeout: 0 lvb_type: 0 Jul 10 16:53:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client dda58b78-27c3-1b63-d778-dfc595795aab (at 10.8.30.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2501a99800, cur 1562802820 expire 1562802670 last 1562802593 Jul 10 16:55:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 16:55:33 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 16:57:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 16:57:14 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 10 17:01:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 17:01:23 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 10 17:02:09 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 17:02:09 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 738 previous similar messages Jul 10 17:02:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 17:02:21 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 10 17:07:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 17:07:25 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 10 17:08:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 17:08:02 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 17:08:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 07c1712c-9739-2dce-4883-ed8d604a7bd1 (at 10.8.15.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f251c28d800, cur 1562803728 expire 1562803578 last 1562803501 Jul 10 17:08:48 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 10 17:08:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 07c1712c-9739-2dce-4883-ed8d604a7bd1 (at 10.8.15.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1a40bdc800, cur 1562803729 expire 1562803579 last 1562803502 Jul 10 17:08:49 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 17:11:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 17:11:28 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 10 17:12:11 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 17:12:11 fir-md1-s1 kernel: LustreError: 21298:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 610 previous similar messages Jul 10 17:12:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 17:12:30 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 10 17:17:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 17:17:37 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 17:22:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 17:22:03 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 17:22:11 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 17:22:11 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 679 previous similar messages Jul 10 17:22:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 59d57404-d19d-2713-ed04-b4a9aba223b9 (at 10.8.25.19@o2ib6) Jul 10 17:22:30 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 10 17:22:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 17:22:31 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 10 17:27:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 17:27:38 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 17:32:15 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 17:32:15 fir-md1-s1 kernel: LustreError: 46562:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 652 previous similar messages Jul 10 17:32:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 17:32:19 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 17:32:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 17:32:43 fir-md1-s1 kernel: Lustre: Skipped 128 previous similar messages Jul 10 17:33:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 10 17:33:04 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 10 17:37:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 17:37:47 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 10 17:42:18 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 17:42:18 fir-md1-s1 kernel: LustreError: 21737:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 577 previous similar messages Jul 10 17:42:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 17:42:46 fir-md1-s1 kernel: Lustre: Skipped 103 previous similar messages Jul 10 17:43:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 17:43:05 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 10 17:47:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 17:47:49 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 10 17:51:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 17:51:16 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 17:52:25 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 17:52:25 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 607 previous similar messages Jul 10 17:52:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 17:52:47 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 10 17:53:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 17:53:11 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 10 17:58:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 17:58:16 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 10 18:02:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 18:02:26 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 18:02:30 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 18:02:30 fir-md1-s1 kernel: LustreError: 44037:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 600 previous similar messages Jul 10 18:02:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 18:02:57 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 10 18:04:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 18:04:06 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 10 18:09:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 18:09:49 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 18:12:33 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 18:12:33 fir-md1-s1 kernel: LustreError: 56756:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 587 previous similar messages Jul 10 18:12:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client bb6c1ebe-228f-c2b0-845a-14ae6de0b327 (at 10.8.27.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24efec2000, cur 1562807578 expire 1562807428 last 1562807351 Jul 10 18:13:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 18:13:04 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 10 18:13:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 18:13:50 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 18:14:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 18:14:50 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 18:20:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 18:20:07 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 18:21:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 214ba30a-c145-16d9-1a66-918c9f83d9e3 (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1e37f0e000, cur 1562808070 expire 1562807920 last 1562807843 Jul 10 18:21:10 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 18:22:36 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 18:22:36 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 622 previous similar messages Jul 10 18:23:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 18:23:10 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 10 18:24:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 18:24:53 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 10 18:26:03 fir-md1-s1 kernel: Lustre: 22005:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1ea5198c00 x1634315021315568/t413200361866(0) o36->a6b91a43-6f67-a7e7-0e97-a87e8033e0cf@10.8.9.10@o2ib6:8/0 lens 488/3152 e 1 to 0 dl 1562808368 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 18:26:18 fir-md1-s1 kernel: Lustre: 23704:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562808366/real 1562808366] req@ffff8f34bdb34b00 x1636729757388096/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562808378 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 10 18:26:18 fir-md1-s1 kernel: Lustre: 23704:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 10 18:26:30 fir-md1-s1 kernel: Lustre: 23704:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562808378/real 1562808378] req@ffff8f34bdb34b00 x1636729757388096/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562808390 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 18:26:31 fir-md1-s1 kernel: Lustre: 23664:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2ed7eb8c00 x1634315021315728/t413200363957(0) o36->a6b91a43-6f67-a7e7-0e97-a87e8033e0cf@10.8.9.10@o2ib6:6/0 lens 488/3152 e 0 to 0 dl 1562808396 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 18:26:42 fir-md1-s1 kernel: LustreError: 23704:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.29.1@o2ib6) failed to reply to blocking AST (req@ffff8f34bdb34b00 x1636729757388096 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f1be7fa6c00/0x5d9ee63987a52a40 lrc: 4/0,0 mode: PR/PR res: [0x200029c10:0xe1b:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db68d4bec9 expref: 1561005 pid: 24581 timeout: 1923479 lvb_type: 0 Jul 10 18:26:42 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.8.29.1@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 10 18:26:42 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 36s: evicting client at 10.8.29.1@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1be7fa6c00/0x5d9ee63987a52a40 lrc: 3/0,0 mode: PR/PR res: [0x200029c10:0xe1b:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db68d4bec9 expref: 1561002 pid: 24581 timeout: 0 lvb_type: 0 Jul 10 18:26:42 fir-md1-s1 kernel: LustreError: 21268:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.1@o2ib6 arrived at 1562808402 with bad export cookie 6746082289101563363 Jul 10 18:26:43 fir-md1-s1 kernel: LustreError: 20368:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.1@o2ib6 arrived at 1562808403 with bad export cookie 6746082289101563363 Jul 10 18:26:43 fir-md1-s1 kernel: LustreError: 20368:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 29 previous similar messages Jul 10 18:26:44 fir-md1-s1 kernel: LustreError: 31003:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.1@o2ib6 arrived at 1562808404 with bad export cookie 6746082289101563363 Jul 10 18:26:44 fir-md1-s1 kernel: LustreError: 31003:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 46 previous similar messages Jul 10 18:26:46 fir-md1-s1 kernel: LustreError: 31011:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.1@o2ib6 arrived at 1562808406 with bad export cookie 6746082289101563363 Jul 10 18:26:46 fir-md1-s1 kernel: LustreError: 31011:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 102 previous similar messages Jul 10 18:26:50 fir-md1-s1 kernel: LustreError: 20371:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.1@o2ib6 arrived at 1562808410 with bad export cookie 6746082289101563363 Jul 10 18:26:50 fir-md1-s1 kernel: LustreError: 20371:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 165 previous similar messages Jul 10 18:26:58 fir-md1-s1 kernel: LustreError: 25075:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.1@o2ib6 arrived at 1562808418 with bad export cookie 6746082289101563363 Jul 10 18:26:58 fir-md1-s1 kernel: LustreError: 25075:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 381 previous similar messages Jul 10 18:27:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 18:27:06 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 18:27:14 fir-md1-s1 kernel: LustreError: 21765:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.1@o2ib6 arrived at 1562808434 with bad export cookie 6746082289101563363 Jul 10 18:27:14 fir-md1-s1 kernel: LustreError: 21765:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 810 previous similar messages Jul 10 18:28:12 fir-md1-s1 kernel: LustreError: 23704:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562808402, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2e9c32b840/0x5d9ee6398d9c6f1f lrc: 3/0,1 mode: --/PW res: [0x200029c10:0xe1b:0x0].0x0 bits 0x2/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23704 timeout: 0 lvb_type: 0 Jul 10 18:28:55 fir-md1-s1 kernel: Lustre: 23617:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f10b4e15400 x1636718262757968/t0(0) o101->1b90433c-235e-7531-cfe6-8ebc9f785a9b@10.9.0.64@o2ib4:0/0 lens 600/3264 e 0 to 0 dl 1562808540 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 18:29:26 fir-md1-s1 kernel: LNet: Service thread pid 23704 was inactive for 200.23s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 10 18:29:26 fir-md1-s1 kernel: Pid: 23704, comm: mdt02_079 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 10 18:29:26 fir-md1-s1 kernel: Call Trace: Jul 10 18:29:26 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 10 18:29:26 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 10 18:29:26 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 10 18:29:26 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 10 18:29:26 fir-md1-s1 kernel: [] mdt_reint_object_lock+0x2c/0x60 [mdt] Jul 10 18:29:26 fir-md1-s1 kernel: [] mdt_reint_striped_lock+0x8c/0x510 [mdt] Jul 10 18:29:26 fir-md1-s1 kernel: [] mdt_reint_setattr+0x6c8/0x1340 [mdt] Jul 10 18:29:26 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jul 10 18:29:26 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jul 10 18:29:26 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jul 10 18:29:26 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 10 18:29:26 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 10 18:29:26 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 10 18:29:26 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 10 18:29:26 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 10 18:29:26 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 10 18:29:26 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1562808566.23704 Jul 10 18:30:00 fir-md1-s1 kernel: LustreError: 23580:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562808510, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f0984291680/0x5d9ee6398e3a7a93 lrc: 3/1,0 mode: --/PR res: [0x200029c10:0xe1b:0x0].0x0 bits 0x13/0x48 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23580 timeout: 0 lvb_type: 0 Jul 10 18:30:12 fir-md1-s1 kernel: LustreError: 21412:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f34f3b1c200 x1636729761310096/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 10 18:30:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a6b91a43-6f67-a7e7-0e97-a87e8033e0cf (at 10.8.9.10@o2ib6) reconnecting Jul 10 18:30:14 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 10 18:30:34 fir-md1-s1 kernel: LustreError: 23601:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f34a2a2c500 x1636729761545232/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 10 18:30:37 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f28722c5100 x1631603052224096/t0(0) o36->40db60e6-2b5f-e52d-2610-43b84e2f829d@10.8.29.1@o2ib6:12/0 lens 488/3152 e 0 to 0 dl 1562808642 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 18:30:41 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.29.1@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1e7bfccc80/0x5d9ee6394b77d89d lrc: 3/0,0 mode: PR/PR res: [0x20002993f:0x1006:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db6257c5f0 expref: 702469 pid: 22005 timeout: 1923701 lvb_type: 0 Jul 10 18:30:54 fir-md1-s1 kernel: LustreError: 23745:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1e5dbe8000 x1636729761785872/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 10 18:31:19 fir-md1-s1 kernel: Lustre: 23664:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2ca9e1a400 x1631603052233024/t413200456057(0) o36->40db60e6-2b5f-e52d-2610-43b84e2f829d@10.8.29.1@o2ib6:24/0 lens 488/3152 e 0 to 0 dl 1562808684 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 18:31:23 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.29.1@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f2e15625340/0x5d9ee6398bc44d4b lrc: 3/0,0 mode: PR/PR res: [0x20002993f:0x181f:0x0].0x0 bits 0x1b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db6af89600 expref: 630078 pid: 23714 timeout: 1923743 lvb_type: 0 Jul 10 18:31:25 fir-md1-s1 kernel: LustreError: 20463:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f2fc5c1ce00 x1636729762168304/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 10 18:31:37 fir-md1-s1 kernel: LustreError: 26258:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f206bb10c00 x1636729762303536/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 10 18:31:42 fir-md1-s1 kernel: LustreError: 21412:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562808612, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2f11c10fc0/0x5d9ee6398ef5837a lrc: 3/0,1 mode: --/PW res: [0x20002993f:0x1006:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21412 timeout: 0 lvb_type: 0 Jul 10 18:31:50 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2ed15e7500 x1631603052238544/t0(0) o101->40db60e6-2b5f-e52d-2610-43b84e2f829d@10.8.29.1@o2ib6:25/0 lens 480/568 e 0 to 0 dl 1562808715 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 18:31:51 fir-md1-s1 kernel: LNet: Service thread pid 23580 was inactive for 200.38s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 10 18:31:51 fir-md1-s1 kernel: Pid: 23580, comm: mdt00_069 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 10 18:31:51 fir-md1-s1 kernel: Call Trace: Jul 10 18:31:51 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 10 18:31:51 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 10 18:31:51 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 10 18:31:51 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 10 18:31:51 fir-md1-s1 kernel: [] mdt_object_lock_try+0x27/0xb0 [mdt] Jul 10 18:31:51 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] Jul 10 18:31:51 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Jul 10 18:31:51 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 10 18:31:51 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 10 18:31:51 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 10 18:31:51 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 10 18:31:51 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 10 18:31:51 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 10 18:31:51 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 10 18:31:51 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 10 18:31:51 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 10 18:31:51 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 10 18:31:51 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1562808711.23580 Jul 10 18:31:54 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.29.1@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f2fcdc2c800/0x5d9ee6398c365e65 lrc: 3/0,0 mode: PW/PW res: [0x20002993f:0x1816:0x0].0x0 bits 0x40/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db6b219522 expref: 580067 pid: 23745 timeout: 1923774 lvb_type: 0 Jul 10 18:32:06 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.29.1@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1fa46c7500/0x5d9ee6398c0e7aed lrc: 3/0,0 mode: PR/PR res: [0x20002993f:0x181c:0x0].0x0 bits 0x1b/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db6b129d27 expref: 561388 pid: 24584 timeout: 1923786 lvb_type: 0 Jul 10 18:32:08 fir-md1-s1 kernel: LustreError: 20463:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f2fc5c1cb00 x1636729762672288/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 10 18:32:24 fir-md1-s1 kernel: LustreError: 23745:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562808654, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2aeefe2880/0x5d9ee6398f2a8ca9 lrc: 3/0,1 mode: --/PW res: [0x20002993f:0x181f:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23745 timeout: 0 lvb_type: 0 Jul 10 18:32:36 fir-md1-s1 kernel: LustreError: 21290:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 18:32:36 fir-md1-s1 kernel: LustreError: 21290:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 487 previous similar messages Jul 10 18:32:37 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.29.1@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f14ff3df2c0/0x5d9ee6398c379831 lrc: 3/0,0 mode: PR/PR res: [0x20002993f:0x1816:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db6b2217c1 expref: 514659 pid: 23750 timeout: 1923817 lvb_type: 0 Jul 10 18:32:52 fir-md1-s1 kernel: Lustre: 23747:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-24), not sending early reply req@ffff8f3416a36c00 x1631603052245472/t413200474285(0) o36->40db60e6-2b5f-e52d-2610-43b84e2f829d@10.8.29.1@o2ib6:27/0 lens 488/3152 e 0 to 0 dl 1562808777 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 18:32:54 fir-md1-s1 kernel: LustreError: 20728:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1ea519da00 x1636729763178096/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 10 18:33:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 0a855284-c89f-aa4a-1498-3c8d9206b44d (at 10.8.9.10@o2ib6) Jul 10 18:33:21 fir-md1-s1 kernel: Lustre: Skipped 136 previous similar messages Jul 10 18:33:23 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.29.1@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f1fed675a00/0x5d9ee639851dd129 lrc: 3/0,0 mode: PR/PR res: [0x200029939:0xb6d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db678b7e95 expref: 448900 pid: 20731 timeout: 1923863 lvb_type: 0 Jul 10 18:33:33 fir-md1-s1 kernel: LNet: Service thread pid 21412 was inactive for 200.36s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 10 18:33:33 fir-md1-s1 kernel: Pid: 21412, comm: mdt02_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 10 18:33:33 fir-md1-s1 kernel: Call Trace: Jul 10 18:33:33 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 10 18:33:33 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 10 18:33:33 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 10 18:33:33 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 10 18:33:33 fir-md1-s1 kernel: [] mdt_reint_object_lock+0x2c/0x60 [mdt] Jul 10 18:33:33 fir-md1-s1 kernel: [] mdt_reint_striped_lock+0x8c/0x510 [mdt] Jul 10 18:33:33 fir-md1-s1 kernel: [] mdt_reint_setattr+0x6c8/0x1340 [mdt] Jul 10 18:33:33 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jul 10 18:33:33 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jul 10 18:33:33 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jul 10 18:33:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 10 18:33:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 10 18:33:33 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 10 18:33:33 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 10 18:33:33 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 10 18:33:33 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 10 18:33:33 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1562808813.21412 Jul 10 18:33:38 fir-md1-s1 kernel: LustreError: 20463:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562808728, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f298b332ac0/0x5d9ee6398f8f2c8d lrc: 3/0,1 mode: --/PW res: [0x20002993f:0x1816:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20463 timeout: 0 lvb_type: 0 Jul 10 18:33:58 fir-md1-s1 kernel: LNet: Service thread pid 21412 completed after 225.75s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jul 10 18:34:01 fir-md1-s1 kernel: LustreError: 23738:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1e5dbeb300 x1636729763835648/t0(0) o104->fir-MDT0000@10.8.29.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 10 18:34:24 fir-md1-s1 kernel: LustreError: 20728:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562808774, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f17c11fe9c0/0x5d9ee6398fc64c5c lrc: 3/0,1 mode: --/EX res: [0x200029939:0xb6d:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20728 timeout: 0 lvb_type: 0 Jul 10 18:34:25 fir-md1-s1 kernel: Lustre: 25675:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f261fb1d400 x1631603052275760/t0(0) o101->40db60e6-2b5f-e52d-2610-43b84e2f829d@10.8.29.1@o2ib6:0/0 lens 1776/3288 e 0 to 0 dl 1562808870 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 18:34:25 fir-md1-s1 kernel: Lustre: 25675:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 10 18:34:30 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.29.1@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f2fe2c85a00/0x5d9ee6397543f6cc lrc: 3/0,0 mode: PR/PR res: [0x200020f18:0x1303e:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.29.1@o2ib6 remote: 0x3ac5b6db6308e65a expref: 359372 pid: 23750 timeout: 1923930 lvb_type: 0 Jul 10 18:35:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 18:35:20 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 10 18:35:28 fir-md1-s1 kernel: LNet: Service thread pid 20463 was inactive for 200.05s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 10 18:35:28 fir-md1-s1 kernel: Pid: 20463, comm: mdt02_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 10 18:35:28 fir-md1-s1 kernel: Call Trace: Jul 10 18:35:28 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 10 18:35:28 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 10 18:35:28 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 10 18:35:28 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 10 18:35:28 fir-md1-s1 kernel: [] mdt_reint_object_lock+0x2c/0x60 [mdt] Jul 10 18:35:28 fir-md1-s1 kernel: [] mdt_reint_striped_lock+0x8c/0x510 [mdt] Jul 10 18:35:28 fir-md1-s1 kernel: [] mdt_reint_setattr+0x6c8/0x1340 [mdt] Jul 10 18:35:28 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jul 10 18:35:28 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jul 10 18:35:28 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jul 10 18:35:28 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 10 18:35:28 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 10 18:35:28 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 10 18:35:28 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 10 18:35:28 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 10 18:35:28 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 10 18:35:28 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1562808928.20463 Jul 10 18:36:11 fir-md1-s1 kernel: LNet: Service thread pid 23704 completed after 605.15s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jul 10 18:36:11 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Jul 10 18:36:32 fir-md1-s1 kernel: LustreError: 24583:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562808902, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f1f99dc7500/0x5d9ee639909a6dfa lrc: 3/0,1 mode: --/PW res: [0x20002993f:0x1817:0x0].0x0 bits 0x40/0x0 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 24583 timeout: 0 lvb_type: 0 Jul 10 18:38:23 fir-md1-s1 kernel: LNet: Service thread pid 24583 was inactive for 200.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 10 18:38:23 fir-md1-s1 kernel: Pid: 24583, comm: mdt01_061 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 10 18:38:23 fir-md1-s1 kernel: Call Trace: Jul 10 18:38:23 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 10 18:38:23 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 10 18:38:23 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 10 18:38:23 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 10 18:38:23 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Jul 10 18:38:23 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Jul 10 18:38:23 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Jul 10 18:38:23 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 10 18:38:23 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 10 18:38:23 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 10 18:38:23 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 10 18:38:23 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 10 18:38:23 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 10 18:38:23 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 10 18:38:23 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 10 18:38:23 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 10 18:38:23 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 10 18:38:23 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1562809103.24583 Jul 10 18:39:04 fir-md1-s1 kernel: LNet: Service thread pid 24583 completed after 241.73s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jul 10 18:40:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 18:40:17 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 10 18:42:48 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 18:42:48 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 671 previous similar messages Jul 10 18:43:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 18:43:23 fir-md1-s1 kernel: Lustre: Skipped 103 previous similar messages Jul 10 18:45:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 18:45:22 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 10 18:46:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 18:46:24 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 18:50:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 18:50:23 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 10 18:52:50 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 18:52:50 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 730 previous similar messages Jul 10 18:53:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 18:53:31 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 10 18:56:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 10 18:56:13 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 10 18:57:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 18:57:46 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 19:00:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 19:00:53 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 19:02:54 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 19:02:54 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 636 previous similar messages Jul 10 19:03:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 19:03:37 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 10 19:06:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 19:06:21 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 10 19:10:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 19:10:59 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 19:12:56 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 19:12:56 fir-md1-s1 kernel: LustreError: 46531:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 747 previous similar messages Jul 10 19:13:38 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 19:13:38 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 10 19:14:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 19:16:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 19:16:25 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 10 19:21:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 19:21:32 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 10 19:23:08 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 19:23:08 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 617 previous similar messages Jul 10 19:23:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 19:23:47 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 10 19:24:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 19:24:53 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 19:26:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 10 19:26:26 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 10 19:31:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 19:31:37 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 10 19:33:10 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 19:33:10 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 747 previous similar messages Jul 10 19:33:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 19:33:54 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 10 19:36:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 19:36:33 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 19:40:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5282ca62-d94c-33fd-9d61-31ebcd98e0af (at 10.9.116.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148dddc400, cur 1562812823 expire 1562812673 last 1562812596 Jul 10 19:40:23 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 19:41:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 19:41:19 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 19:41:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 19:41:53 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 19:43:12 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 19:43:12 fir-md1-s1 kernel: LustreError: 46524:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 667 previous similar messages Jul 10 19:43:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 19:43:59 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 10 19:46:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 19:46:55 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 10 19:51:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 19:52:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 19:52:07 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 19:53:14 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 19:53:14 fir-md1-s1 kernel: LustreError: 46522:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 656 previous similar messages Jul 10 19:54:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 19:54:06 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 10 19:56:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 19:56:56 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 10 20:01:49 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562814102/real 1562814102] req@ffff8f2ed6939800 x1636729822352496/t0(0) o104->fir-MDT0002@10.9.115.13@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562814109 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 10 20:01:49 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 10 20:01:56 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562814109/real 1562814109] req@ffff8f2ed6939800 x1636729822352496/t0(0) o104->fir-MDT0002@10.9.115.13@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562814116 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 20:01:57 fir-md1-s1 kernel: Lustre: 23728:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2f1953a700 x1631603054418480/t0(0) o101->40db60e6-2b5f-e52d-2610-43b84e2f829d@10.8.29.1@o2ib6:2/0 lens 1784/3288 e 1 to 0 dl 1562814122 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 20:01:57 fir-md1-s1 kernel: Lustre: 23728:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 10 20:02:03 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562814116/real 1562814116] req@ffff8f2ed6939800 x1636729822352496/t0(0) o104->fir-MDT0002@10.9.115.13@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562814123 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 20:02:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 10 20:02:09 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 10 20:02:17 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562814130/real 1562814130] req@ffff8f2ed6939800 x1636729822352496/t0(0) o104->fir-MDT0002@10.9.115.13@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562814137 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 20:02:17 fir-md1-s1 kernel: Lustre: 21003:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 10 20:02:17 fir-md1-s1 kernel: LustreError: 21003:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.115.13@o2ib4) failed to reply to blocking AST (req@ffff8f2ed6939800 x1636729822352496 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f0a0118a1c0/0x5d9ee639a3c8f1ab lrc: 4/0,0 mode: PR/PR res: [0x2c002c3a1:0xcc22:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.115.13@o2ib4 remote: 0xb3c53fda679b06b6 expref: 1833 pid: 23555 timeout: 1929219 lvb_type: 0 Jul 10 20:02:17 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.9.115.13@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 10 20:02:17 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.9.115.13@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f0a0118a1c0/0x5d9ee639a3c8f1ab lrc: 3/0,0 mode: PR/PR res: [0x2c002c3a1:0xcc22:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.115.13@o2ib4 remote: 0xb3c53fda679b06b6 expref: 1834 pid: 23555 timeout: 0 lvb_type: 0 Jul 10 20:02:17 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jul 10 20:03:16 fir-md1-s1 kernel: LustreError: 81718:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 20:03:16 fir-md1-s1 kernel: LustreError: 81718:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 662 previous similar messages Jul 10 20:04:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 20:04:07 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 10 20:05:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 19195713-2529-2820-f0a1-33d24d172ab7 (at 10.9.115.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2a71eab800, cur 1562814319 expire 1562814169 last 1562814092 Jul 10 20:05:19 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 10 20:07:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 20:07:00 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 20:08:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 20:08:36 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 20:12:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 20:12:16 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 10 20:13:19 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 20:13:19 fir-md1-s1 kernel: LustreError: 46510:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 658 previous similar messages Jul 10 20:14:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 20:14:11 fir-md1-s1 kernel: Lustre: Skipped 116 previous similar messages Jul 10 20:19:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 20:19:16 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 10 20:19:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 20:19:37 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 20:22:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 20:22:17 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 20:23:21 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 20:23:21 fir-md1-s1 kernel: LustreError: 46553:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 620 previous similar messages Jul 10 20:24:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 20:24:20 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 10 20:29:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 20:29:17 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 20:31:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 20:31:19 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 20:32:34 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 0a2c93c8-5b84-dccd-112a-6823da10a94a (at 10.9.116.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f07d77b0000, cur 1562815954 expire 1562815804 last 1562815727 Jul 10 20:32:34 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 10 20:32:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 20:32:49 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 10 20:33:24 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 20:33:24 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 474 previous similar messages Jul 10 20:34:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 20:34:23 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 10 20:39:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 20:39:21 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 10 20:42:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 20:42:36 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 10 20:43:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 20:43:06 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 10 20:43:24 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 20:43:24 fir-md1-s1 kernel: LustreError: 79335:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 522 previous similar messages Jul 10 20:44:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 20:44:24 fir-md1-s1 kernel: Lustre: Skipped 130 previous similar messages Jul 10 20:49:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 20:49:25 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 10 20:52:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 20:52:40 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 20:53:26 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 20:53:26 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 555 previous similar messages Jul 10 20:53:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 20:53:28 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 20:54:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 20:54:25 fir-md1-s1 kernel: Lustre: Skipped 108 previous similar messages Jul 10 21:02:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 21:02:02 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 10 21:03:32 fir-md1-s1 kernel: LustreError: 22059:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 21:03:32 fir-md1-s1 kernel: LustreError: 22059:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 574 previous similar messages Jul 10 21:04:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 21:04:19 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 21:04:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 21:04:44 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 10 21:05:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 21:05:15 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 10 21:13:48 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 21:13:48 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 590 previous similar messages Jul 10 21:14:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 21:14:09 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 10 21:14:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 21:14:22 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 21:15:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 21:15:33 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 21:15:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 21:15:49 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 10 21:21:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2503a3ec00, cur 1562818910 expire 1562818760 last 1562818683 Jul 10 21:23:50 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 21:23:50 fir-md1-s1 kernel: LustreError: 21544:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 589 previous similar messages Jul 10 21:24:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 21:24:26 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 10 21:26:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 21:26:08 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 21:26:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 21:26:08 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 10 21:26:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 21:26:18 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 21:33:53 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 10 21:33:53 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 619 previous similar messages Jul 10 21:34:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 21:34:27 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 21:36:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 21:36:10 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 21:36:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 21:36:10 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 10 21:38:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 21:43:13 fir-md1-s1 kernel: Lustre: 22007:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562820186/real 1562820186] req@ffff8f162cf13900 x1636729886578256/t0(0) o104->fir-MDT0002@10.8.16.5@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562820193 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 10 21:43:18 fir-md1-s1 kernel: Lustre: 23621:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562820191/real 1562820191] req@ffff8f369fa1b600 x1636729886638064/t0(0) o104->fir-MDT0002@10.8.16.5@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562820198 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 10 21:43:21 fir-md1-s1 kernel: Lustre: 22006:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1d509fb600 x1638719331699472/t0(0) o101->957c1ad0-d547-b44d-0f14-5f92c3213a3d@10.8.15.3@o2ib6:26/0 lens 1784/3288 e 1 to 0 dl 1562820206 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 21:43:25 fir-md1-s1 kernel: Lustre: 23621:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562820198/real 1562820198] req@ffff8f369fa1b600 x1636729886638064/t0(0) o104->fir-MDT0002@10.8.16.5@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562820205 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 21:43:25 fir-md1-s1 kernel: Lustre: 23621:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 10 21:43:26 fir-md1-s1 kernel: Lustre: 21423:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3c16bf4800 x1638711356333600/t0(0) o101->524f09b9-37f3-6401-947e-a803ba6b2d1e@10.9.114.5@o2ib4:1/0 lens 1784/3288 e 1 to 0 dl 1562820211 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 21:43:34 fir-md1-s1 kernel: Lustre: 23617:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562820207/real 1562820207] req@ffff8f0c80f14b00 x1636729886710368/t0(0) o104->fir-MDT0002@10.8.16.5@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562820214 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 10 21:43:34 fir-md1-s1 kernel: Lustre: 23617:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 10 21:43:41 fir-md1-s1 kernel: LustreError: 22007:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.16.5@o2ib6) failed to reply to blocking AST (req@ffff8f162cf13900 x1636729886578256 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f1a93b30b40/0x5d9ee639a990f2f5 lrc: 4/0,0 mode: PR/PR res: [0x2c002c3a1:0xc980:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.16.5@o2ib6 remote: 0xe66f7278c9b52327 expref: 3088 pid: 97661 timeout: 1935303 lvb_type: 0 Jul 10 21:43:41 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.16.5@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 10 21:43:41 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.16.5@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f1a93b30b40/0x5d9ee639a990f2f5 lrc: 3/0,0 mode: PR/PR res: [0x2c002c3a1:0xc980:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.16.5@o2ib6 remote: 0xe66f7278c9b52327 expref: 3089 pid: 97661 timeout: 0 lvb_type: 0 Jul 10 21:44:03 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 21:44:03 fir-md1-s1 kernel: LustreError: 21514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 672 previous similar messages Jul 10 21:44:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 21:44:40 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 10 21:46:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 21:46:20 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 21:46:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 21:46:20 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 10 21:46:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 1fb1c1bc-a5c2-7639-1248-10341b490c82 (at 10.8.16.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34ec9dbc00, cur 1562820397 expire 1562820247 last 1562820170 Jul 10 21:49:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 21:49:05 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 21:54:08 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 21:54:08 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 588 previous similar messages Jul 10 21:54:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 10 21:54:56 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 10 21:56:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 21:56:31 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 10 21:56:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 10 21:56:31 fir-md1-s1 kernel: Lustre: Skipped 102 previous similar messages Jul 10 21:58:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2fcc7c4000, cur 1562821134 expire 1562820984 last 1562820907 Jul 10 21:58:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 10 22:00:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 22:00:21 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 22:04:10 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 22:04:10 fir-md1-s1 kernel: LustreError: 21683:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 649 previous similar messages Jul 10 22:06:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 22:06:31 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 10 22:06:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 22:06:39 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 22:06:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 22:06:39 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 10 22:11:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 22:11:02 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 10 22:13:37 fir-md1-s1 kernel: Lustre: 23754:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2ee0d44e00 x1638719625554688/t0(0) o101->957c1ad0-d547-b44d-0f14-5f92c3213a3d@10.8.15.3@o2ib6:12/0 lens 376/1600 e 1 to 0 dl 1562822022 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 22:13:51 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.15.3@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f0640def2c0/0x5d9ee639bdc0fe27 lrc: 3/0,0 mode: PR/PR res: [0x2c002c443:0x17f:0x0].0x0 bits 0x5b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.8.15.3@o2ib6 remote: 0xc36b1972b7953a19 expref: 204 pid: 23586 timeout: 1937091 lvb_type: 0 Jul 10 22:13:51 fir-md1-s1 kernel: LustreError: 23738:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f1e08449800 ns: mdt-fir-MDT0002_UUID lock: ffff8f1460fad7c0/0x5d9ee639bdc0ffb6 lrc: 3/0,0 mode: EX/EX res: [0x2c002c443:0x17f:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT flags: 0x50000000000000 nid: 10.8.15.3@o2ib6 remote: 0xc36b1972b7953a27 expref: 116 pid: 23738 timeout: 0 lvb_type: 3 Jul 10 22:13:51 fir-md1-s1 kernel: Lustre: 23738:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:9s); client may timeout. req@ffff8f2ee0d44e00 x1638719625554688/t351778161854(0) o101->957c1ad0-d547-b44d-0f14-5f92c3213a3d@10.8.15.3@o2ib6:12/0 lens 376/1568 e 1 to 0 dl 1562822022 ref 1 fl Complete:/0/0 rc -107/-107 Jul 10 22:14:11 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 22:14:11 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 634 previous similar messages Jul 10 22:15:57 fir-md1-s1 kernel: Lustre: 10502:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562822149/real 1562822149] req@ffff8f0db72aef00 x1636729907279168/t0(0) o104->fir-MDT0002@10.9.113.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562822156 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 10 22:15:57 fir-md1-s1 kernel: Lustre: 10502:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 10 22:16:04 fir-md1-s1 kernel: Lustre: 26258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562822156/real 1562822156] req@ffff8f1d0705b000 x1636729907279552/t0(0) o104->fir-MDT0002@10.9.113.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562822163 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 22:16:04 fir-md1-s1 kernel: Lustre: 26258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 10 22:16:04 fir-md1-s1 kernel: Lustre: 10198:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0e36a3fb00 x1635100579630112/t0(0) o101->81c79b6e-3061-2fda-8521-bc0b462e4ff6@10.9.113.13@o2ib4:9/0 lens 1784/3288 e 1 to 0 dl 1562822169 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 22:16:05 fir-md1-s1 kernel: Lustre: 23708:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0c20893600 x1635100579630176/t0(0) o101->81c79b6e-3061-2fda-8521-bc0b462e4ff6@10.9.113.13@o2ib4:10/0 lens 1784/3288 e 1 to 0 dl 1562822170 ref 2 fl Interpret:/0/0 rc 0/0 Jul 10 22:16:11 fir-md1-s1 kernel: Lustre: 10502:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562822164/real 1562822164] req@ffff8f0db72aef00 x1636729907279168/t0(0) o104->fir-MDT0002@10.9.113.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562822171 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 22:16:11 fir-md1-s1 kernel: Lustre: 10502:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 10 22:16:25 fir-md1-s1 kernel: Lustre: 26258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562822178/real 1562822178] req@ffff8f1d0705b000 x1636729907279552/t0(0) o104->fir-MDT0002@10.9.113.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562822185 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 10 22:16:25 fir-md1-s1 kernel: LustreError: 10502:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.113.3@o2ib4) failed to reply to blocking AST (req@ffff8f0db72aef00 x1636729907279168 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f20c7640900/0x5d9ee639b06fbfb3 lrc: 4/0,0 mode: PR/PR res: [0x2c002c3a1:0xcb52:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.113.3@o2ib4 remote: 0xf46aef741591092c expref: 4407 pid: 21482 timeout: 1937267 lvb_type: 0 Jul 10 22:16:25 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.9.113.3@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 10 22:16:25 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 36s: evicting client at 10.9.113.3@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f20c7640900/0x5d9ee639b06fbfb3 lrc: 3/0,0 mode: PR/PR res: [0x2c002c3a1:0xcb52:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.113.3@o2ib4 remote: 0xf46aef741591092c expref: 4408 pid: 21482 timeout: 0 lvb_type: 0 Jul 10 22:16:25 fir-md1-s1 kernel: Lustre: 26258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 10 22:16:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 10 22:16:40 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 22:16:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 22:16:40 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 10 22:18:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 22:18:00 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 10 22:18:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 0f8f808f-b03b-81e6-e30e-46ff547f2e45 (at 10.9.113.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fd205000, cur 1562822336 expire 1562822186 last 1562822109 Jul 10 22:24:15 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 22:24:15 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 639 previous similar messages Jul 10 22:26:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 22:26:42 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 10 22:26:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 22:26:42 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 10 22:27:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 22:27:39 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 10 22:29:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 22:29:01 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 10 22:34:17 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 22:34:17 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 715 previous similar messages Jul 10 22:36:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 22:36:48 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 10 22:36:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 10 22:36:48 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 10 22:39:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 22:39:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 22:39:24 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 10 22:39:24 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 10 22:44:19 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 22:44:19 fir-md1-s1 kernel: LustreError: 24213:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 624 previous similar messages Jul 10 22:46:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 22:46:51 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 10 22:47:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 22:47:00 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 22:50:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 22:50:03 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 10 22:54:21 fir-md1-s1 kernel: LustreError: 21617:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 22:54:21 fir-md1-s1 kernel: LustreError: 21617:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 600 previous similar messages Jul 10 22:56:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 22:56:51 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 10 22:56:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 22:56:56 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 22:57:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 22:57:02 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 10 23:00:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 23:00:07 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 10 23:04:23 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 23:04:23 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 696 previous similar messages Jul 10 23:07:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 10 23:07:00 fir-md1-s1 kernel: Lustre: Skipped 107 previous similar messages Jul 10 23:07:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 23:07:13 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 10 23:10:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 23:10:41 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 10 23:12:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 23:12:00 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 10 23:14:25 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 23:14:25 fir-md1-s1 kernel: LustreError: 21708:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 663 previous similar messages Jul 10 23:17:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 23:17:05 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 10 23:17:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 23:17:20 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 10 23:20:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 23:20:47 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 10 23:22:38 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client cc0184b4-423e-d61b-ff8b-e62121180b57 (at 10.9.113.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f153524d400, cur 1562826158 expire 1562826008 last 1562825931 Jul 10 23:22:38 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 10 23:22:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 92ffa420-d747-a973-baf2-68cec64e7e81 (at 10.9.113.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1488f49400, cur 1562826160 expire 1562826010 last 1562825933 Jul 10 23:22:40 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 10 23:23:54 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 153 seconds. I think it's dead, and I am evicting it. exp ffff8f3ae6b90000, cur 1562826234 expire 1562826084 last 1562826081 Jul 10 23:24:27 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 23:24:27 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 617 previous similar messages Jul 10 23:25:10 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client dd74c27b-57ee-efbf-9952-be3ffdfb9c30 (at 10.9.114.4@o2ib4) in 175 seconds. I think it's dead, and I am evicting it. exp ffff8f252eb47400, cur 1562826310 expire 1562826160 last 1562826135 Jul 10 23:25:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d4242da5-5a9c-4508-f9da-c1e7f36347f4 (at 10.9.114.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f252173d000, cur 1562826348 expire 1562826198 last 1562826121 Jul 10 23:25:48 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 10 23:26:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d4242da5-5a9c-4508-f9da-c1e7f36347f4 (at 10.9.114.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f45182c8800, cur 1562826362 expire 1562826212 last 1562826135 Jul 10 23:26:02 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 10 23:27:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 23:27:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 23:27:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 23:27:07 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 23:27:07 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 23:27:07 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 10 23:27:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 10 23:27:44 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 10 23:34:30 fir-md1-s1 kernel: LustreError: 46560:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 23:34:30 fir-md1-s1 kernel: LustreError: 46560:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 618 previous similar messages Jul 10 23:36:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 23:36:21 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 10 23:37:11 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3c16ae5000, cur 1562827031 expire 1562826881 last 1562826804 Jul 10 23:37:11 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 10 23:37:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 10 23:37:15 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 10 23:37:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 10 23:37:46 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 10 23:44:34 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 23:44:34 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 529 previous similar messages Jul 10 23:46:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 10 23:46:27 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 10 23:47:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 23:47:19 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 10 23:47:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 23:47:47 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 10 23:47:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 23:47:58 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 10 23:49:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 23:52:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 10 23:52:23 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 10 23:54:36 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 10 23:54:36 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 465 previous similar messages Jul 10 23:54:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 8f13c09d-70f2-3426-bcdb-b5b12d23066d (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f5bb8000, cur 1562828088 expire 1562827938 last 1562827861 Jul 10 23:55:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 8f13c09d-70f2-3426-bcdb-b5b12d23066d (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34f27cb400, cur 1562828102 expire 1562827952 last 1562827875 Jul 10 23:55:02 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 10 23:57:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 10 23:57:02 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 10 23:57:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 10 23:57:19 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 10 23:57:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 10 23:57:53 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 10 23:58:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 00:04:41 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 00:04:41 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 498 previous similar messages Jul 11 00:07:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 00:07:32 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 11 00:08:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 11 00:08:00 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 00:08:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 00:08:15 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 11 00:14:42 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 00:14:42 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 483 previous similar messages Jul 11 00:17:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 00:17:37 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 11 00:18:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 00:18:13 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 00:18:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 00:18:37 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 11 00:24:43 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 00:24:43 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 524 previous similar messages Jul 11 00:27:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 00:27:53 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 11 00:28:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 00:28:20 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 11 00:29:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 00:29:04 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 00:33:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 00:33:51 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 00:34:46 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 00:34:46 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 557 previous similar messages Jul 11 00:37:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 00:37:56 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 11 00:38:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 11 00:38:29 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 11 00:39:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 00:39:43 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 00:42:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 00:44:47 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 00:44:47 fir-md1-s1 kernel: LustreError: 21485:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 549 previous similar messages Jul 11 00:47:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 00:47:56 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 11 00:49:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 00:49:02 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 00:49:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 00:49:28 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 00:50:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 00:50:23 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 11 00:54:49 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 11 00:54:49 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 642 previous similar messages Jul 11 00:58:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 00:58:10 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 01:00:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 01:00:49 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 11 01:00:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 01:00:57 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 11 01:04:57 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 01:04:57 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 607 previous similar messages Jul 11 01:05:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:05:29 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 01:07:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:08:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 01:08:19 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 11 01:09:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:10:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 01:10:54 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 11 01:11:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 01:11:20 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 01:15:00 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 01:15:00 fir-md1-s1 kernel: LustreError: 46535:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 607 previous similar messages Jul 11 01:15:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:18:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 01:18:28 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 11 01:20:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 01:20:57 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 11 01:23:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 01:23:01 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 01:25:03 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 01:25:03 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 664 previous similar messages Jul 11 01:28:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 01:28:31 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 11 01:31:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 01:31:55 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 11 01:32:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:32:04 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 01:33:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:33:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 01:33:31 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 01:35:10 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 01:35:10 fir-md1-s1 kernel: LustreError: 25630:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 549 previous similar messages Jul 11 01:35:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f198ef57400, cur 1562834151 expire 1562834001 last 1562833924 Jul 11 01:38:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 01:38:37 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 11 01:39:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:42:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 01:42:05 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 11 01:42:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:44:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 01:44:01 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 11 01:45:12 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 01:45:12 fir-md1-s1 kernel: LustreError: 22670:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 578 previous similar messages Jul 11 01:47:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:47:55 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 01:48:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 01:48:58 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 11 01:52:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 01:52:06 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 11 01:54:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 01:54:02 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 01:55:15 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 01:55:15 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 500 previous similar messages Jul 11 01:59:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 01:59:07 fir-md1-s1 kernel: Lustre: Skipped 108 previous similar messages Jul 11 01:59:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 01:59:16 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 02:02:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 02:02:26 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 11 02:04:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 02:04:08 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 11 02:05:17 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 02:05:17 fir-md1-s1 kernel: LustreError: 46545:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 500 previous similar messages Jul 11 02:09:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 02:09:11 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 11 02:11:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 02:11:53 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 02:13:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 02:13:01 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 11 02:14:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 02:14:14 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 11 02:15:25 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 11 02:15:25 fir-md1-s1 kernel: LustreError: 20505:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 507 previous similar messages Jul 11 02:19:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 02:19:15 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 11 02:22:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 02:22:01 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 02:23:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 02:23:12 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 11 02:24:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 02:24:40 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 02:25:25 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 02:25:25 fir-md1-s1 kernel: LustreError: 21685:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 639 previous similar messages Jul 11 02:29:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 02:29:36 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 11 02:33:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 02:33:01 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 02:33:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 02:33:24 fir-md1-s1 kernel: Lustre: Skipped 41087 previous similar messages Jul 11 02:35:26 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 02:35:26 fir-md1-s1 kernel: LustreError: 21711:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 627 previous similar messages Jul 11 02:36:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 02:36:35 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 02:37:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24cb033c00, cur 1562837824 expire 1562837674 last 1562837597 Jul 11 02:39:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 02:39:42 fir-md1-s1 kernel: Lustre: Skipped 41111 previous similar messages Jul 11 02:43:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 02:43:48 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 11 02:45:31 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 94208 GRANT, real grant 0 Jul 11 02:45:31 fir-md1-s1 kernel: LustreError: 46542:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 555 previous similar messages Jul 11 02:48:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 02:48:02 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 11 02:49:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 02:49:46 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 02:54:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 02:54:06 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 02:55:39 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 02:55:39 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 535 previous similar messages Jul 11 02:58:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client b9620cc9-0642-09f3-d857-9cdbad9511de (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2533fe6400, cur 1562839101 expire 1562838951 last 1562838874 Jul 11 02:58:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 02:58:24 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 11 02:58:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 02:58:50 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 11 02:59:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 02:59:49 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 03:03:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 03:04:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 03:04:06 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 03:05:41 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 03:05:41 fir-md1-s1 kernel: LustreError: 22429:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 641 previous similar messages Jul 11 03:05:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 03:08:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 03:08:53 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 03:09:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 03:09:56 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 11 03:11:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 03:11:43 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 03:14:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 03:14:18 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 03:15:44 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 03:15:44 fir-md1-s1 kernel: LustreError: 21292:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 558 previous similar messages Jul 11 03:20:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 11 03:20:02 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 03:20:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 03:20:02 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 11 03:22:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 03:22:05 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 03:24:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 03:24:20 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 03:24:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client cac1eba7-cdaa-957f-8735-d5169807717b (at 10.9.112.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3c662b5800, cur 1562840685 expire 1562840535 last 1562840458 Jul 11 03:24:45 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 11 03:25:45 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 03:25:45 fir-md1-s1 kernel: LustreError: 46541:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 509 previous similar messages Jul 11 03:30:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 03:30:25 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 03:30:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 03:30:25 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 11 03:32:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 03:32:59 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 03:35:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 03:35:02 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 03:35:52 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 28672 GRANT, real grant 0 Jul 11 03:35:52 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 537 previous similar messages Jul 11 03:40:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 03:40:25 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 03:42:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 03:42:42 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 11 03:43:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 03:43:01 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 11 03:45:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 03:45:32 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 11 03:45:56 fir-md1-s1 kernel: LustreError: 46560:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 03:45:56 fir-md1-s1 kernel: LustreError: 46560:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 554 previous similar messages Jul 11 03:50:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 03:50:28 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 11 03:53:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 03:53:30 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 03:54:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 03:54:53 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 11 03:55:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 03:55:35 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 03:55:58 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 03:55:58 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 504 previous similar messages Jul 11 04:00:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 04:00:39 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 11 04:04:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 74771de5-63d2-ad4d-0853-e29847bc9774 (at 10.9.116.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2d1d67c800, cur 1562843064 expire 1562842914 last 1562842837 Jul 11 04:04:24 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 11 04:04:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 74771de5-63d2-ad4d-0853-e29847bc9774 (at 10.9.116.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f22d6423c00, cur 1562843079 expire 1562842929 last 1562842852 Jul 11 04:04:39 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 11 04:05:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 04:05:38 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 04:05:58 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 04:05:58 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 391 previous similar messages Jul 11 04:06:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 04:06:24 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 04:06:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 04:06:50 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 04:09:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 92a5fc1a-0f67-1260-3d67-1ac1c4c2c6d6 (at 10.8.28.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25215c6400, cur 1562843396 expire 1562843246 last 1562843169 Jul 11 04:10:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 04:10:55 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 11 04:16:01 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 04:16:01 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 463 previous similar messages Jul 11 04:16:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 04:16:15 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 04:17:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 04:17:19 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 04:18:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 04:18:46 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 11 04:21:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 04:21:05 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 11 04:26:08 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 657250be-d5db-acec-954e-1239d7463eca claims 155648 GRANT, real grant 0 Jul 11 04:26:08 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 393 previous similar messages Jul 11 04:27:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 04:27:13 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 11 04:27:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 04:27:52 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 04:29:17 fir-md1-s1 kernel: Lustre: 50444:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1bfea3bc00 x1631639284829952/t0(0) o36->e18301fc-f860-0db4-bf24-6c606e0cc839@10.8.8.31@o2ib6:22/0 lens 520/2888 e 1 to 0 dl 1562844562 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:29:24 fir-md1-s1 kernel: Lustre: 22288:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1c4ecb4800 x1638201205215568/t0(0) o36->5a22b190-14d6-e96a-6855-6cd3296f5726@10.9.104.72@o2ib4:29/0 lens 568/2888 e 1 to 0 dl 1562844569 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:29:29 fir-md1-s1 kernel: Lustre: 23729:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f44a3b27800 x1634080641904016/t0(0) o36->c6e3bcd8-71de-d683-20ac-e6684b91d659@10.9.108.10@o2ib4:4/0 lens 600/2888 e 0 to 0 dl 1562844574 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:29:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 04:29:32 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 11 04:29:34 fir-md1-s1 kernel: Lustre: 23726:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f3fecb89e00 x1631580975217936/t0(0) o36->f070aa79-4085-01c4-e45c-5c90a853bda7@10.9.106.25@o2ib4:9/0 lens 552/2888 e 0 to 0 dl 1562844579 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:29:42 fir-md1-s1 kernel: Lustre: 22007:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1daa4fb600 x1631548097618640/t0(0) o36->a62b9648-73d4-4e84-cbc3-4dd2cc8c6b56@10.9.106.24@o2ib4:17/0 lens 568/2888 e 1 to 0 dl 1562844587 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:30:04 fir-md1-s1 kernel: Lustre: 22007:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1de7e53600 x1634130859459504/t0(0) o36->190e8c90-938d-b7f6-84df-7662b8e78e53@10.9.107.71@o2ib4:9/0 lens 584/2888 e 1 to 0 dl 1562844609 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:30:04 fir-md1-s1 kernel: Lustre: 22007:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 11 04:30:32 fir-md1-s1 kernel: Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Jul 11 04:30:32 fir-md1-s1 kernel: LustreError: 97661:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562844542, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8f24ff7aad00/0x5d9ee63a066764a5 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT flags: 0x1000001000000 nid: local remote: 0x5d9ee63a066764ac expref: -99 pid: 97661 timeout: 0 lvb_type: 0 Jul 11 04:30:34 fir-md1-s1 kernel: LustreError: 23663:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562844544, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8f3d49bec380/0x5d9ee63a06688438 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT flags: 0x1000001000000 nid: local remote: 0x5d9ee63a0668843f expref: -99 pid: 23663 timeout: 0 lvb_type: 0 Jul 11 04:30:39 fir-md1-s1 kernel: LustreError: 23587:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562844549, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8f393cabe300/0x5d9ee63a066ae688 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT flags: 0x1000001000000 nid: local remote: 0x5d9ee63a066ae68f expref: -99 pid: 23587 timeout: 0 lvb_type: 0 Jul 11 04:30:45 fir-md1-s1 kernel: Lustre: 20728:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2308d54b00 x1633774881944864/t0(0) o36->bb635275-94e4-0a1a-209a-677b17ce9a5a@10.9.104.50@o2ib4:20/0 lens 552/2888 e 0 to 0 dl 1562844650 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:30:45 fir-md1-s1 kernel: Lustre: 20728:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 11 04:30:57 fir-md1-s1 kernel: LustreError: 50444:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562844567, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8f0a01e12400/0x5d9ee63a067236ad lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT flags: 0x1000001000000 nid: local remote: 0x5d9ee63a067236b4 expref: -99 pid: 50444 timeout: 0 lvb_type: 0 Jul 11 04:30:57 fir-md1-s1 kernel: LustreError: 50444:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 11 04:31:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 8172217c-cb28-d209-5f1f-4aceb1d4d3a6 (at 10.8.8.31@o2ib6) Jul 11 04:31:08 fir-md1-s1 kernel: Lustre: Skipped 104 previous similar messages Jul 11 04:31:10 fir-md1-s1 kernel: LustreError: 26258:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562844580, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8f1bf44318c0/0x5d9ee63a0678cfe4 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 14 type: IBT flags: 0x1000001000000 nid: local remote: 0x5d9ee63a0678cfeb expref: -99 pid: 26258 timeout: 0 lvb_type: 0 Jul 11 04:31:10 fir-md1-s1 kernel: LustreError: 26258:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 11 04:31:20 fir-md1-s1 kernel: LustreError: 20725:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562844589, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8f064baa7500/0x5d9ee63a067caf60 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 14 type: IBT flags: 0x1000001000000 nid: local remote: 0x5d9ee63a067caf67 expref: -99 pid: 20725 timeout: 0 lvb_type: 0 Jul 11 04:31:20 fir-md1-s1 kernel: LustreError: 20725:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 11 04:31:25 fir-md1-s1 kernel: Lustre: 23750:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2b12e88600 x1638083510157968/t0(0) o36->5fc014af-e3d7-51ad-6083-2ba5cb7bd6c2@10.9.114.8@o2ib4:0/0 lens 584/2888 e 0 to 0 dl 1562844690 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:31:25 fir-md1-s1 kernel: Lustre: 23750:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 11 04:37:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 04:37:24 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 11 04:37:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 04:37:58 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 11 04:39:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 04:39:43 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 11 04:41:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 04:41:13 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 11 04:47:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 04:47:26 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 11 04:48:12 fir-md1-s1 kernel: Lustre: 23614:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562845685/real 1562845685] req@ffff8f1bfea3c500 x1636730140589504/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562845692 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 04:48:19 fir-md1-s1 kernel: Lustre: 23614:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562845692/real 1562845692] req@ffff8f1bfea3c500 x1636730140589504/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562845699 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 04:48:20 fir-md1-s1 kernel: Lustre: 21675:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f442ca18c00 x1636296525541360/t0(0) o36->f7eae5f9-18e9-99eb-0207-24a1fdf92451@10.9.113.2@o2ib4:25/0 lens 488/3152 e 1 to 0 dl 1562845705 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 04:48:26 fir-md1-s1 kernel: Lustre: 23614:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562845699/real 1562845699] req@ffff8f1bfea3c500 x1636730140589504/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562845706 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 04:48:40 fir-md1-s1 kernel: Lustre: 23614:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562845713/real 1562845713] req@ffff8f1bfea3c500 x1636730140589504/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562845720 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 04:48:40 fir-md1-s1 kernel: Lustre: 23614:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 04:48:40 fir-md1-s1 kernel: LustreError: 23614:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.112.17@o2ib4) failed to reply to blocking AST (req@ffff8f1bfea3c500 x1636730140589504 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f2f04b05a00/0x5d9ee639b06374d1 lrc: 4/0,0 mode: PR/PR res: [0x20002985c:0x475:0x0].0x0 bits 0x1b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.9.112.17@o2ib4 remote: 0x8809107949661ab4 expref: 28 pid: 23628 timeout: 1960802 lvb_type: 0 Jul 11 04:48:40 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.9.112.17@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 11 04:48:40 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.9.112.17@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff8f2f04b05a00/0x5d9ee639b06374d1 lrc: 3/0,0 mode: PR/PR res: [0x20002985c:0x475:0x0].0x0 bits 0x1b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.9.112.17@o2ib4 remote: 0x8809107949661ab4 expref: 29 pid: 23628 timeout: 0 lvb_type: 0 Jul 11 04:48:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 04:48:57 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 04:49:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 04:49:53 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 11 04:51:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 04:51:28 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 11 04:51:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ae10cc76-adf2-6fa2-11b9-b27d5e4703ab (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fd379000, cur 1562845904 expire 1562845754 last 1562845677 Jul 11 04:51:44 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 11 04:57:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 04:57:29 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 04:59:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 04:59:00 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 11 05:01:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 05:01:01 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 11 05:01:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 05:01:33 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 11 05:08:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 05:08:12 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 11 05:09:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 05:09:16 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 05:11:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 05:11:08 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 11 05:11:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 05:11:36 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 11 05:18:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 05:18:17 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 05:21:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 05:21:12 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 11 05:21:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 05:21:25 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 05:21:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 05:21:37 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 11 05:28:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 05:28:18 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 05:31:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 05:31:19 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 05:31:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 05:31:31 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 05:31:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 05:31:37 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 11 05:38:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 05:38:24 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 05:41:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 05:41:40 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 11 05:41:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 05:41:40 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 11 05:42:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 05:42:36 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 05:46:56 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jul 11 05:46:56 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Jul 11 05:48:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 05:48:40 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 11 05:51:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 05:51:57 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 11 05:51:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 11 05:51:57 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 11 05:55:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 05:55:20 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 05:58:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 05:58:47 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 06:02:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 06:02:02 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Jul 11 06:04:35 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0d626ed000, cur 1562850275 expire 1562850125 last 1562850048 Jul 11 06:04:35 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 11 06:05:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 06:05:52 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 11 06:07:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 06:07:21 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 11 06:08:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 06:08:48 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 06:12:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 06:12:21 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 11 06:16:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 11 06:16:10 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 11 06:19:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 06:19:11 fir-md1-s1 kernel: Lustre: Skipped 445323 previous similar messages Jul 11 06:19:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 06:19:17 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 06:23:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 06:23:22 fir-md1-s1 kernel: Lustre: Skipped 445373 previous similar messages Jul 11 06:25:06 fir-md1-s1 kernel: Lustre: 23572:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562851499/real 1562851499] req@ffff8f0d034ad700 x1636730185487712/t0(0) o106->fir-MDT0000@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562851506 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 06:25:13 fir-md1-s1 kernel: Lustre: 23572:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562851506/real 1562851506] req@ffff8f0d034ad700 x1636730185487712/t0(0) o106->fir-MDT0000@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562851513 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 06:25:14 fir-md1-s1 kernel: Lustre: 10588:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f09d5525d00 x1637044331667008/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:19/0 lens 480/568 e 1 to 0 dl 1562851519 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 06:25:20 fir-md1-s1 kernel: Lustre: 23572:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562851513/real 1562851513] req@ffff8f0d034ad700 x1636730185487712/t0(0) o106->fir-MDT0000@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562851520 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 06:25:20 fir-md1-s1 kernel: Lustre: 23572:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:1s); client may timeout. req@ffff8f09d5525d00 x1637044331667008/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:19/0 lens 480/536 e 1 to 0 dl 1562851519 ref 1 fl Complete:/0/0 rc 301/301 Jul 11 06:26:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 06:26:14 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 06:29:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 06:29:28 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 11 06:29:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 06:29:29 fir-md1-s1 kernel: Lustre: Skipped 116 previous similar messages Jul 11 06:33:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 06:33:33 fir-md1-s1 kernel: Lustre: Skipped 164 previous similar messages Jul 11 06:36:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 06:36:34 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 11 06:39:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 06:39:31 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 11 06:39:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 06:39:35 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 06:40:32 fir-md1-s1 kernel: Lustre: 20465:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2e006d8900 x1631353224125856/t0(0) o101->17e26c1e-4877-4fff-89e1-78bf5463918b@10.8.11.6@o2ib6:7/0 lens 376/1600 e 1 to 0 dl 1562852437 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 06:41:47 fir-md1-s1 kernel: LustreError: 23627:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562852417, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2f0c68d100/0x5d9ee63a16136f3f lrc: 3/0,1 mode: --/EX res: [0x200029d48:0x1:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23627 timeout: 0 lvb_type: 0 Jul 11 06:42:17 fir-md1-s1 kernel: Lustre: 23627:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (82:38s); client may timeout. req@ffff8f2e006d8900 x1631353224125856/t413216701507(0) o101->17e26c1e-4877-4fff-89e1-78bf5463918b@10.8.11.6@o2ib6:7/0 lens 376/1568 e 1 to 0 dl 1562852499 ref 1 fl Complete:/0/0 rc 0/0 Jul 11 06:43:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 06:43:36 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 11 06:47:55 fir-md1-s1 kernel: Lustre: 23708:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562852867/real 1562852867] req@ffff8f0e9f8c2100 x1636730195096224/t0(0) o106->fir-MDT0000@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562852874 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 06:47:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 06:47:55 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 11 06:49:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 06:49:42 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 11 06:52:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 06:52:21 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 11 06:53:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 06:53:42 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 11 06:58:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 06:58:05 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 11 07:00:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 07:00:29 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 07:02:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 07:02:55 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 07:03:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 07:03:45 fir-md1-s1 kernel: Lustre: Skipped 115 previous similar messages Jul 11 07:08:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 07:08:13 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 11 07:10:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 07:10:38 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 07:13:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 07:13:46 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 11 07:14:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 07:14:04 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 07:18:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 11 07:18:33 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 07:20:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 07:20:53 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 11 07:24:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 07:24:05 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 11 07:25:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 07:25:27 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 11 07:29:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 07:29:22 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 07:31:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 07:31:15 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 11 07:34:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 07:34:06 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 11 07:36:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 07:36:53 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 07:39:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 11 07:39:22 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 11 07:41:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 07:41:28 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 07:44:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 07:44:12 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 11 07:47:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 07:47:13 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 11 07:50:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 07:50:53 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 11 07:51:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 07:51:43 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 11 07:54:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 07:54:26 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 11 07:58:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 07:58:22 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 08:02:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 08:02:01 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 11 08:02:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 08:02:14 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 11 08:04:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 08:04:37 fir-md1-s1 kernel: Lustre: Skipped 109 previous similar messages Jul 11 08:08:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 08:08:42 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 08:12:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 08:12:02 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 11 08:12:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 11 08:12:19 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 11 08:14:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 08:14:37 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 11 08:21:35 fir-md1-s1 kernel: Lustre: 23751:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562858488/real 1562858488] req@ffff8f2647e9b600 x1636730239955200/t0(0) o104->fir-MDT0000@10.8.11.6@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562858495 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 08:22:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 08:22:49 fir-md1-s1 kernel: Lustre: Skipped 181 previous similar messages Jul 11 08:23:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 08:23:08 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 11 08:23:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 08:23:34 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 08:24:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 08:24:38 fir-md1-s1 kernel: Lustre: Skipped 218 previous similar messages Jul 11 08:32:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 08:32:52 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 11 08:34:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 08:34:40 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 11 08:35:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 08:35:15 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 08:35:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 11 08:35:43 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 08:38:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client aa133483-5248-262d-e748-a147c987e0e5 (at 10.9.108.65@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1506141c00, cur 1562859494 expire 1562859344 last 1562859267 Jul 11 08:42:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 08:42:59 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 11 08:44:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 08:44:41 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 11 08:45:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 08:45:52 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 08:45:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 08:45:55 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 08:53:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 08:53:43 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 08:54:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 08:54:43 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 11 08:55:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 08:55:58 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 11 08:56:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 08:56:32 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 11 08:57:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9eb449c2-e54f-1e34-81bc-f024b214ecc1 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f18eddc1400, cur 1562860639 expire 1562860489 last 1562860412 Jul 11 08:57:19 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 11 08:57:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9eb449c2-e54f-1e34-81bc-f024b214ecc1 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4536050400, cur 1562860649 expire 1562860499 last 1562860422 Jul 11 08:57:29 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 11 09:03:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 09:03:52 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 09:05:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 09:05:05 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 11 09:07:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 09:07:23 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 11 09:07:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 09:07:50 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 11 09:11:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2c74246800, cur 1562861476 expire 1562861326 last 1562861249 Jul 11 09:11:16 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 11 09:14:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 09:14:16 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 09:15:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 09:15:09 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 11 09:17:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 3528ecaf-52ef-a9ab-e1d8-8a0bbcc53063 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2f92f8dc00, cur 1562861865 expire 1562861715 last 1562861638 Jul 11 09:17:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 3528ecaf-52ef-a9ab-e1d8-8a0bbcc53063 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1b7b501800, cur 1562861871 expire 1562861721 last 1562861644 Jul 11 09:17:51 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 11 09:18:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 09:18:14 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 09:19:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client cd3d0230-3738-e2d9-7e9f-2fd94c27579a (at 10.9.115.5@o2ib4) in 160 seconds. I think it's dead, and I am evicting it. exp ffff8f251b635400, cur 1562861941 expire 1562861791 last 1562861781 Jul 11 09:19:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client cd3d0230-3738-e2d9-7e9f-2fd94c27579a (at 10.9.115.5@o2ib4) in 166 seconds. I think it's dead, and I am evicting it. exp ffff8f4518ea2c00, cur 1562861947 expire 1562861797 last 1562861781 Jul 11 09:19:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 11 09:19:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 09:19:57 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 11 09:21:26 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 11 09:21:26 fir-md1-s1 kernel: LustreError: 22430:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 108 previous similar messages Jul 11 09:24:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 09:24:24 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 09:25:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 09:25:21 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 11 09:26:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client e389cf81-f921-5b21-a2d2-508161f0a482 (at 10.9.114.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1e7a75e400, cur 1562862398 expire 1562862248 last 1562862171 Jul 11 09:29:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 09:29:45 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 11 09:31:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 09:31:34 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 09:34:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 09:34:27 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 11 09:35:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 09:35:22 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 11 09:41:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 09:41:47 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 11 09:42:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 09:42:14 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 11 09:44:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 09:44:32 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 09:45:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 09:45:48 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 11 09:46:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 683c6245-5f05-50dc-7e48-4fd959186454 (at 10.9.114.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2ecf6cdc00, cur 1562863593 expire 1562863443 last 1562863366 Jul 11 09:46:33 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 11 09:54:43 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f12d1bee000, cur 1562864083 expire 1562863933 last 1562863856 Jul 11 09:54:43 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 11 09:54:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 09:54:47 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 11 09:54:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 11 09:54:54 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 09:55:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 09:55:23 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 09:55:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 09:55:54 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 11 10:04:26 fir-md1-s1 kernel: LustreError: 22059:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 11 10:04:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 10:04:50 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 11 10:06:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 10:06:05 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 11 10:07:33 fir-md1-s1 kernel: Lustre: 23618:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864845/real 1562864845] req@ffff8f15350b1500 x1636730289756624/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864852 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 10:07:33 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864845/real 1562864845] req@ffff8f0f2d2bbf00 x1636730289756592/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864852 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 10:07:33 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 10:07:40 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864852/real 1562864852] req@ffff8f0cc475a400 x1636730289756576/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864859 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 10:07:40 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 10:07:40 fir-md1-s1 kernel: Lustre: 23703:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1299bf8600 x1637045642245168/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:15/0 lens 480/568 e 1 to 0 dl 1562864865 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 10:07:47 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864859/real 1562864859] req@ffff8f0f2d2bbf00 x1636730289756592/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864866 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 10:07:47 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 10:07:54 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864867/real 1562864867] req@ffff8f0f2d2bbf00 x1636730289756592/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864874 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 10:07:54 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 11 10:08:01 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864874/real 1562864874] req@ffff8f0cc475a400 x1636730289756576/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864881 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 10:08:01 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 10:08:15 fir-md1-s1 kernel: Lustre: 23618:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864888/real 1562864888] req@ffff8f15350b1500 x1636730289756624/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864895 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 10:08:15 fir-md1-s1 kernel: Lustre: 23618:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 11 10:08:36 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864909/real 1562864909] req@ffff8f0cc475a400 x1636730289756576/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864916 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 10:08:36 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562864909/real 1562864909] req@ffff8f0f2d2bbf00 x1636730289756592/t0(0) o106->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562864916 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 10:08:36 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 11 10:08:36 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 10:09:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 88cac62b-9ed1-f52c-09d1-c83e30477915 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f6a8c4c00, cur 1562864941 expire 1562864791 last 1562864714 Jul 11 10:09:01 fir-md1-s1 kernel: Lustre: 23568:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:76s); client may timeout. req@ffff8f0527b35100 x1637045642245184/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:15/0 lens 480/536 e 1 to 0 dl 1562864865 ref 1 fl Complete:/0/0 rc 301/301 Jul 11 10:09:01 fir-md1-s1 kernel: Lustre: 23568:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages Jul 11 10:12:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 10:12:12 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 11 10:15:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 10:15:00 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 11 10:16:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 10:16:29 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 11 10:20:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 10:20:45 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 11 10:23:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 10:23:13 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 11 10:26:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 10:26:14 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 11 10:26:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 10:26:33 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 11 10:32:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 10:33:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 10:33:31 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 10:35:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 10:36:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 10:36:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 10:36:31 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 10:36:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 10:36:40 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 11 10:43:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 10:44:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 10:44:30 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 11 10:46:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 10:46:41 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 10:46:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 10:46:41 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 11 10:49:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 10:49:10 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 10:50:35 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli 1522cbeb-4bdd-6d96-7026-321415672330 claims 28672 GRANT, real grant 0 Jul 11 10:54:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 10:54:53 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 11 10:56:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 10:56:44 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 11 10:57:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 10:57:19 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 10:58:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 10:58:15 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 11:05:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 11:05:04 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 11 11:06:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 11:06:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 11:06:50 fir-md1-s1 kernel: Lustre: Skipped 111 previous similar messages Jul 11 11:07:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 11:07:24 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 11 11:16:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 11:16:52 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 11 11:17:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 11:17:13 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 11 11:17:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 11:17:37 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 11:17:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 11:17:38 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 11:26:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 11:26:53 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 11 11:27:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 11:27:26 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 11:28:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 11:28:15 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 11:33:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 11:33:59 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 11:37:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 11:37:24 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 11 11:37:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 11:37:36 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 11:38:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 11:38:16 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 11 11:46:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 11:46:45 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 11:47:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 11:47:28 fir-md1-s1 kernel: Lustre: Skipped 124 previous similar messages Jul 11 11:47:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 11:47:46 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 11 11:48:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 11:48:40 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 11:49:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9964d113-fbb3-bb3d-6283-d900df7d14b0 (at 10.9.106.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2502d76c00, cur 1562870941 expire 1562870791 last 1562870714 Jul 11 11:49:01 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 11 11:49:02 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 78d0a867-f444-fefc-cf10-2f40e2381985 (at 10.9.106.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14ebfda400, cur 1562870942 expire 1562870792 last 1562870715 Jul 11 11:49:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9964d113-fbb3-bb3d-6283-d900df7d14b0 (at 10.9.106.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34f2428800, cur 1562870946 expire 1562870796 last 1562870719 Jul 11 11:50:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 606c5bb8-0e42-25d6-4ebe-304dc77c2b78 (at 10.9.115.4@o2ib4) in 180 seconds. I think it's dead, and I am evicting it. exp ffff8f251fd04800, cur 1562871017 expire 1562870867 last 1562870837 Jul 11 11:50:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 606c5bb8-0e42-25d6-4ebe-304dc77c2b78 (at 10.9.115.4@o2ib4) in 187 seconds. I think it's dead, and I am evicting it. exp ffff8f34fdb37c00, cur 1562871022 expire 1562870872 last 1562870835 Jul 11 11:50:22 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 11 11:50:22 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562871015/real 1562871015] req@ffff8f0a06665d00 x1636730350774576/t0(0) o106->fir-MDT0000@10.9.106.11@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562871022 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 11:50:22 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 11 11:50:30 fir-md1-s1 kernel: Lustre: 23572:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0b92bf7500 x1637046316264608/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:5/0 lens 480/568 e 1 to 0 dl 1562871035 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 11:50:30 fir-md1-s1 kernel: Lustre: 23572:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 11 11:50:36 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562871029/real 1562871029] req@ffff8f0a06665d00 x1636730350774576/t0(0) o106->fir-MDT0000@10.9.106.11@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562871036 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 11:50:36 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 11:50:57 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562871050/real 1562871050] req@ffff8f0a06665d00 x1636730350774576/t0(0) o106->fir-MDT0000@10.9.106.11@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562871057 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 11:50:57 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 11 11:51:32 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562871085/real 1562871085] req@ffff8f0a06665d00 x1636730350774576/t0(0) o106->fir-MDT0000@10.9.106.11@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562871092 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 11:51:32 fir-md1-s1 kernel: Lustre: 10506:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 11 11:51:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client b482f036-7d17-2da6-47f4-65a7cfc97276 (at 10.9.115.6@o2ib4) in 201 seconds. I think it's dead, and I am evicting it. exp ffff8f10e1d1e800, cur 1562871093 expire 1562870943 last 1562870892 Jul 11 11:51:59 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b66b5931-c739-bfc8-7870-ebb3b0803c4b (at 10.9.115.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f07dd387000, cur 1562871119 expire 1562870969 last 1562870892 Jul 11 11:51:59 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 11 11:52:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 8cb89dc0-c88e-79a3-15bf-ba0c55574ada (at 10.9.106.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148b06f800, cur 1562871176 expire 1562871026 last 1562870949 Jul 11 11:52:56 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 11 11:55:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 2c00b5b0-7c71-de91-4f51-5cb7b8de22c7 (at 10.9.112.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3d43af8400, cur 1562871337 expire 1562871187 last 1562871110 Jul 11 11:57:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 11:57:36 fir-md1-s1 kernel: Lustre: Skipped 111 previous similar messages Jul 11 11:57:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 11:57:49 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 11 11:58:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 11:58:57 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 12:02:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 2d5099ae-445e-cb63-4a68-05c28b456049 (at 10.9.112.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d6ce4c000, cur 1562871767 expire 1562871617 last 1562871540 Jul 11 12:02:47 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 11 12:07:27 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client ae2db849-228e-9fc3-9658-c094e0066e91 (at 10.9.102.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f14c7771000, cur 1562872047 expire 1562871897 last 1562871820 Jul 11 12:07:27 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 11 12:07:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 12:07:37 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 11 12:07:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 12:07:39 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 12:08:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 12:08:17 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 12:09:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 12:09:00 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 12:10:01 fir-md1-s1 kernel: Lustre: 23651:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562872194/real 1562872194] req@ffff8f0b6d0ec200 x1636730360125824/t0(0) o104->fir-MDT0002@10.8.28.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562872201 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 12:10:09 fir-md1-s1 kernel: Lustre: 23556:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0ed8ed7200 x1638778776139328/t0(0) o101->61f27ed9-3774-ff36-a4d4-c75cfa800da4@10.9.113.14@o2ib4:14/0 lens 1784/3288 e 1 to 0 dl 1562872214 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 12:10:09 fir-md1-s1 kernel: Lustre: 97651:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562872202/real 1562872202] req@ffff8f1967e24500 x1636730360127776/t0(0) o104->fir-MDT0002@10.8.28.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562872209 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 12:10:09 fir-md1-s1 kernel: Lustre: 97651:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 11 12:10:29 fir-md1-s1 kernel: Lustre: 23651:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562872222/real 1562872222] req@ffff8f0b6d0ec200 x1636730360125824/t0(0) o104->fir-MDT0002@10.8.28.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562872229 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 12:10:29 fir-md1-s1 kernel: Lustre: 23651:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 11 12:10:58 fir-md1-s1 kernel: Lustre: 10198:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0964ab8300 x1634132492115616/t0(0) o101->89c5b213-fa16-71ad-d5f3-58d49989ce10@10.9.115.11@o2ib4:3/0 lens 1784/3288 e 1 to 0 dl 1562872263 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 12:10:58 fir-md1-s1 kernel: Lustre: 10198:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 11 12:11:02 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562872255/real 1562872255] req@ffff8f13d1dc1b00 x1636730360218672/t0(0) o104->fir-MDT0002@10.8.28.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562872262 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 12:11:02 fir-md1-s1 kernel: Lustre: 23560:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Jul 11 12:11:06 fir-md1-s1 kernel: Lustre: 23594:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f1455254b00 x1638788830172848/t0(0) o101->ca693efe-e963-3124-a59d-0beac55f4de3@10.9.112.17@o2ib4:11/0 lens 1784/3288 e 0 to 0 dl 1562872271 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 12:11:06 fir-md1-s1 kernel: Lustre: 23594:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jul 11 12:11:08 fir-md1-s1 kernel: Lustre: 23584:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2a7df04500 x1638779383139440/t0(0) o101->927ebcad-3373-a003-8433-ef313bb0111b@10.8.15.9@o2ib6:13/0 lens 1784/3288 e 0 to 0 dl 1562872273 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 12:11:08 fir-md1-s1 kernel: Lustre: 23584:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Jul 11 12:12:07 fir-md1-s1 kernel: Lustre: 23716:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562872320/real 1562872320] req@ffff8f2efdecaa00 x1636730360221808/t0(0) o104->fir-MDT0002@10.8.28.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562872327 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 12:12:07 fir-md1-s1 kernel: Lustre: 20722:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562872320/real 1562872320] req@ffff8f1e85cff800 x1636730360221616/t0(0) o104->fir-MDT0002@10.8.28.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562872327 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 12:12:07 fir-md1-s1 kernel: Lustre: 20722:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 138 previous similar messages Jul 11 12:12:07 fir-md1-s1 kernel: Lustre: 23716:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 12:12:28 fir-md1-s1 kernel: LustreError: 23651:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.28.1@o2ib6) failed to reply to blocking AST (req@ffff8f0b6d0ec200 x1636730360125824 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f2ef68aa640/0x5d9ee63a3c50834b lrc: 4/0,0 mode: PR/PR res: [0x2c002c3a1:0xde1a:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.28.1@o2ib6 remote: 0xe501a40d47942476 expref: 2646 pid: 23601 timeout: 1987550 lvb_type: 0 Jul 11 12:12:28 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.28.1@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 11 12:12:28 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.28.1@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2ef68aa640/0x5d9ee63a3c50834b lrc: 3/0,0 mode: PR/PR res: [0x2c002c3a1:0xde1a:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.28.1@o2ib6 remote: 0xe501a40d47942476 expref: 2647 pid: 23601 timeout: 0 lvb_type: 0 Jul 11 12:12:28 fir-md1-s1 kernel: Lustre: 23757:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (104:1s); client may timeout. req@ffff8f2eb226cb00 x1638778791357328/t351854447442(0) o101->9623626c-0b75-9f88-dbc1-9e0f1a45143d@10.9.114.4@o2ib4:3/0 lens 1784/1240 e 1 to 0 dl 1562872347 ref 1 fl Complete:/0/0 rc 0/0 Jul 11 12:12:28 fir-md1-s1 kernel: Lustre: 23757:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jul 11 12:17:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 12:17:45 fir-md1-s1 kernel: Lustre: Skipped 128 previous similar messages Jul 11 12:18:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 12:18:22 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 11 12:19:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 12:19:08 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 11 12:19:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 12:19:25 fir-md1-s1 kernel: Lustre: 21410:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562872758/real 1562872758] req@ffff8f14a62aa400 x1636730364103360/t0(0) o106->fir-MDT0002@10.9.103.12@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562872765 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 12:19:25 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562872758/real 1562872758] req@ffff8f0c72e6a400 x1636730364103376/t0(0) o106->fir-MDT0002@10.9.103.12@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562872765 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 12:19:25 fir-md1-s1 kernel: Lustre: 23568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 12:19:25 fir-md1-s1 kernel: Lustre: 21410:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 11 12:19:43 fir-md1-s1 kernel: Lustre: 10501:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f12443f3c00 x1637046440268528/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:18/0 lens 480/568 e 0 to 0 dl 1562872788 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 12:22:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client b21cb12d-36f3-6903-28db-2805bc9f940b (at 10.9.103.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f050405a400, cur 1562872928 expire 1562872778 last 1562872701 Jul 11 12:22:08 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Jul 11 12:22:25 fir-md1-s1 kernel: Lustre: 10588:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (178:9s); client may timeout. req@ffff8f12443f3c00 x1637046440268528/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:18/0 lens 480/536 e 0 to 0 dl 1562872936 ref 1 fl Complete:/0/0 rc 301/301 Jul 11 12:22:25 fir-md1-s1 kernel: Lustre: 10588:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2 previous similar messages Jul 11 12:23:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 12:24:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 12:27:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 12:27:47 fir-md1-s1 kernel: Lustre: Skipped 114 previous similar messages Jul 11 12:28:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 12:28:24 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 11 12:29:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 12:29:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 12:29:10 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 12:36:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 12:37:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 12:37:50 fir-md1-s1 kernel: Lustre: Skipped 109 previous similar messages Jul 11 12:39:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 12:39:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 11 12:39:14 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 11 12:39:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 12:39:39 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 12:47:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 12:47:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 12:47:54 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 11 12:49:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 12:49:31 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 11 12:49:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 12:49:44 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 12:58:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 12:58:18 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 11 13:00:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 13:00:06 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 11 13:00:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 13:00:21 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 13:03:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 13:03:30 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 13:08:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 13:08:18 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 13:10:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 13:10:09 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 13:10:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 13:10:23 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 13:16:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34e6224c00, cur 1562876190 expire 1562876040 last 1562875963 Jul 11 13:16:30 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jul 11 13:16:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 13:16:45 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 13:19:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 13:19:08 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 11 13:20:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 13:20:10 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 13:20:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 13:20:26 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 11 13:29:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 13:29:28 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 11 13:29:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5a80badc-02de-d0da-16f2-dd5cc4f34700 (at 10.9.113.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d3560a800, cur 1562876982 expire 1562876832 last 1562876755 Jul 11 13:30:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 13:30:10 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 11 13:30:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 13:30:32 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 13:30:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 13:30:34 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 13:39:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 13:39:28 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 11 13:40:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 13:40:17 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 13:40:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 13:40:55 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 11 13:43:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 13:49:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 13:49:36 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 11 13:50:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 13:50:19 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 11 13:50:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 13:50:56 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 11 13:59:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 13:59:39 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 11 14:00:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 14:00:20 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 14:00:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 14:00:59 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 14:05:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 14:05:31 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 14:08:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 14:10:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 14:10:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 14:10:01 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 14:10:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 14:11:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 14:11:09 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 11 14:12:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 14:12:05 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 11 14:20:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 14:20:25 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 14:20:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25004d2800, cur 1562880029 expire 1562879879 last 1562879802 Jul 11 14:20:29 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 11 14:21:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 14:21:22 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 14:22:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 14:22:19 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 14:23:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 14:31:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 14:31:07 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 11 14:31:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 14:31:31 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 14:32:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 14:32:22 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 11 14:37:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 14:37:56 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 11 14:39:22 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 14:41:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 14:41:33 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 14:41:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 14:41:33 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 11 14:46:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 14:46:25 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 11 14:48:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 14:48:35 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 14:51:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 14:51:48 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 11 14:51:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 14:51:58 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 14:56:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 14:56:55 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 11 14:58:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 14:58:54 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 15:01:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 15:01:49 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 11 15:02:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 15:02:15 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 15:06:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 15:06:55 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 11 15:12:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 15:12:03 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 11 15:12:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 15:12:31 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 15:13:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 15:13:29 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 15:17:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 15:17:22 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 15:22:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 15:22:04 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jul 11 15:23:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 15:23:00 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 11 15:26:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 15:26:55 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 15:28:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 15:28:16 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 11 15:32:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 15:32:07 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 11 15:33:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 15:33:21 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 15:38:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 15:38:12 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 15:38:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 15:38:27 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 11 15:42:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 15:42:07 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 11 15:43:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 15:43:30 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 11 15:48:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 15:48:47 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 11 15:49:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 15:49:20 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 15:52:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 15:52:26 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 11 15:53:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 15:53:42 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 15:58:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 11 15:58:49 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 11 16:02:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 16:02:31 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 11 16:03:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 16:03:48 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 16:06:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 16:06:30 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 16:08:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 16:08:54 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 11 16:12:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 16:12:56 fir-md1-s1 kernel: Lustre: Skipped 127 previous similar messages Jul 11 16:13:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 16:13:49 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 11 16:18:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 16:18:45 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 16:19:54 fir-md1-s1 kernel: Lustre: 23651:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562887187/real 1562887187] req@ffff8f0769bee600 x1636730486625216/t0(0) o104->fir-MDT0002@10.8.8.22@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562887194 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 16:19:54 fir-md1-s1 kernel: Lustre: 23651:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 75 previous similar messages Jul 11 16:20:02 fir-md1-s1 kernel: Lustre: 23594:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f06ff261e00 x1631602292001840/t0(0) o101->8c191431-c80e-a99c-d724-6274df7fd787@10.9.102.10@o2ib4:7/0 lens 1792/3288 e 1 to 0 dl 1562887207 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 16:20:02 fir-md1-s1 kernel: Lustre: 23594:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 11 16:20:03 fir-md1-s1 kernel: Lustre: 23756:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f26518a6f00 x1631567502066816/t0(0) o101->dacb83f0-b432-ea21-cf1b-fb1ac63fd0b0@10.9.101.62@o2ib4:8/0 lens 576/3264 e 1 to 0 dl 1562887208 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 16:20:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 16:20:05 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 11 16:20:22 fir-md1-s1 kernel: LustreError: 23651:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.8.22@o2ib6) failed to reply to blocking AST (req@ffff8f0769bee600 x1636730486625216 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f2ee5d85e80/0x5d9ee63a9becb427 lrc: 4/0,0 mode: PR/PR res: [0x2c0000404:0x479:0x0].0x0 bits 0x13/0x0 rrc: 44 type: IBT flags: 0x60200400000020 nid: 10.8.8.22@o2ib6 remote: 0xadd31f436d5659b6 expref: 24 pid: 97668 timeout: 2002304 lvb_type: 0 Jul 11 16:20:22 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.8.22@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 11 16:20:22 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.8.22@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f2ee5d85e80/0x5d9ee63a9becb427 lrc: 3/0,0 mode: PR/PR res: [0x2c0000404:0x479:0x0].0x0 bits 0x13/0x0 rrc: 44 type: IBT flags: 0x60200400000020 nid: 10.8.8.22@o2ib6 remote: 0xadd31f436d5659b6 expref: 25 pid: 97668 timeout: 0 lvb_type: 0 Jul 11 16:22:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c0afed87-894c-bd68-b6a7-ca4f7af5df99 (at 10.9.103.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1478a93800, cur 1562887356 expire 1562887206 last 1562887129 Jul 11 16:22:44 fir-md1-s1 kernel: Lustre: 21410:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562887357/real 1562887357] req@ffff8f0f6dfed400 x1636730487568944/t0(0) o106->fir-MDT0000@10.8.15.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562887364 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 16:22:44 fir-md1-s1 kernel: Lustre: 21410:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 11 16:22:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 16:22:57 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 11 16:23:02 fir-md1-s1 kernel: Lustre: 23573:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f0fec7ac200 x1637048159181168/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:7/0 lens 480/568 e 0 to 0 dl 1562887387 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 16:23:02 fir-md1-s1 kernel: Lustre: 23573:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 11 16:23:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 05c0106b-58e3-3894-4263-1c25034da8ce (at 10.8.15.6@o2ib6) in 178 seconds. I think it's dead, and I am evicting it. exp ffff8f2253775c00, cur 1562887432 expire 1562887282 last 1562887254 Jul 11 16:23:52 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jul 11 16:23:52 fir-md1-s1 kernel: Lustre: 21410:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:45s); client may timeout. req@ffff8f0fec7ac200 x1637048159181168/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:7/0 lens 480/536 e 0 to 0 dl 1562887387 ref 1 fl Complete:/0/0 rc 301/301 Jul 11 16:24:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 16:24:20 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 11 16:24:41 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client ada255a7-6ae1-daa7-1ada-5fa3d62ccfb9 (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2d18712c00, cur 1562887481 expire 1562887331 last 1562887254 Jul 11 16:24:41 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 11 16:25:57 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 5afbb881-550f-fa08-cafd-4158b37c9811 (at 10.8.24.16@o2ib6) in 198 seconds. I think it's dead, and I am evicting it. exp ffff8f34eb3aa800, cur 1562887557 expire 1562887407 last 1562887359 Jul 11 16:26:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ac9cd631-a534-1fba-753c-5069b079d1ad (at 10.8.24.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f450560b400, cur 1562887586 expire 1562887436 last 1562887359 Jul 11 16:30:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 16:30:09 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 16:30:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 16:30:33 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 11 16:33:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 16:33:26 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 11 16:34:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 16:34:30 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 16:42:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 16:42:06 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 11 16:42:54 fir-md1-s1 kernel: Lustre: 10501:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 16:42:54 fir-md1-s1 kernel: Lustre: 10501:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1369 previous similar messages Jul 11 16:43:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 16:43:18 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 16:43:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 16:43:35 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 16:44:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 16:44:42 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 16:44:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1c8996d000, cur 1562888693 expire 1562888543 last 1562888466 Jul 11 16:44:53 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 11 16:52:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 16:52:13 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 11 16:53:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 16:53:46 fir-md1-s1 kernel: Lustre: Skipped 106 previous similar messages Jul 11 16:54:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 16:54:44 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 16:54:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 16:54:53 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 17:03:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 17:03:01 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 11 17:03:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 17:03:46 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 11 17:05:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 17:05:00 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 17:06:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 17:06:51 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 17:13:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 11 17:13:04 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 11 17:13:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 17:13:46 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 11 17:15:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 17:15:08 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 11 17:17:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 17:17:00 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 17:18:35 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 17:20:50 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 17:22:00 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 17:23:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 17:23:12 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 11 17:24:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 17:24:07 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 11 17:25:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 17:25:29 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 11 17:28:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 17:28:44 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 11 17:34:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 17:34:03 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 11 17:34:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 17:34:13 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 11 17:35:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 17:35:42 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 17:40:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 17:40:22 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 17:43:43 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 17:43:43 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Jul 11 17:45:18 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 17:45:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 24ab177c-fa53-1ad7-a4b8-75ee3a88aec0 (at 10.8.8.24@o2ib6) Jul 11 17:45:25 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 11 17:45:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 11 17:45:30 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 17:46:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 17:46:37 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 17:46:38 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 17:51:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 17:51:07 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 17:55:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 17:55:27 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 11 17:55:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 17:55:38 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 11 17:57:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 17:57:19 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 11 17:57:50 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 17:58:10 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 18:01:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 18:01:07 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 11 18:05:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 18:05:30 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 11 18:05:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 18:05:51 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 18:06:03 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 18:07:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 18:07:43 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 11 18:09:23 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 18:12:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 18:12:31 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 18:13:48 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 18:15:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 18:15:34 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 11 18:15:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 18:15:51 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 11 18:17:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 18:17:49 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 11 18:23:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 18:23:19 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 11 18:25:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 18:25:37 fir-md1-s1 kernel: Lustre: Skipped 111 previous similar messages Jul 11 18:25:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 18:25:51 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 11 18:28:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 18:28:05 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 18:34:24 fir-md1-s1 kernel: Lustre: 10501:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 18:36:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 18:36:04 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 11 18:38:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 18:38:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 18:38:20 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 18:38:20 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 11 18:40:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 18:40:22 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 18:46:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 18:46:12 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 11 18:48:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 18:48:27 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 11 18:49:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 18:49:02 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 18:51:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 18:51:10 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 18:56:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 18:56:50 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 11 18:58:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 18:58:50 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 11 18:59:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 18:59:12 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 11 19:04:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 19:04:44 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 11 19:06:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 19:06:56 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 11 19:09:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 19:09:08 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 11 19:09:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 19:09:38 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 11 19:17:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 19:17:44 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 11 19:19:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 19:19:13 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 11 19:19:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 19:19:16 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 19:19:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 19:19:44 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 11 19:27:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 19:27:59 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 11 19:29:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 19:29:18 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 11 19:29:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 19:29:46 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 11 19:30:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 19:30:45 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 19:38:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 19:38:11 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 11 19:39:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 19:39:19 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 11 19:40:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 19:40:15 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 11 19:43:12 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 19:43:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 19:43:32 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 19:44:47 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 19:48:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 19:48:20 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 11 19:50:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 11 19:50:19 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 11 19:51:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 19:51:41 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 19:52:12 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 19:55:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 19:55:20 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 19:55:33 fir-md1-s1 kernel: Lustre: 10196:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:55:33 fir-md1-s1 kernel: Lustre: 10196:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 55 previous similar messages Jul 11 19:55:34 fir-md1-s1 kernel: Lustre: 10196:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:55:34 fir-md1-s1 kernel: Lustre: 10196:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message Jul 11 19:55:35 fir-md1-s1 kernel: Lustre: 23568:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:55:35 fir-md1-s1 kernel: Lustre: 23568:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages Jul 11 19:55:37 fir-md1-s1 kernel: Lustre: 10501:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:55:37 fir-md1-s1 kernel: Lustre: 10501:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages Jul 11 19:55:41 fir-md1-s1 kernel: Lustre: 10588:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:55:41 fir-md1-s1 kernel: Lustre: 10588:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 75 previous similar messages Jul 11 19:55:49 fir-md1-s1 kernel: Lustre: 23651:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:55:49 fir-md1-s1 kernel: Lustre: 23651:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 105 previous similar messages Jul 11 19:56:05 fir-md1-s1 kernel: Lustre: 23568:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:56:05 fir-md1-s1 kernel: Lustre: 23568:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 321 previous similar messages Jul 11 19:56:38 fir-md1-s1 kernel: Lustre: 23591:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:56:38 fir-md1-s1 kernel: Lustre: 23591:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1194 previous similar messages Jul 11 19:57:42 fir-md1-s1 kernel: Lustre: 23651:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 19:57:42 fir-md1-s1 kernel: Lustre: 23651:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 63707 previous similar messages Jul 11 19:58:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 19:58:47 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 11 19:59:37 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:00:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 20:00:21 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 20:01:47 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:01:47 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Jul 11 20:04:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 11 20:04:22 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 11 20:06:34 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:08:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1bcba60800, cur 1562900886 expire 1562900736 last 1562900659 Jul 11 20:08:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 20:08:48 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 11 20:10:36 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:10:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 545f12c1-4799-a254-b9c4-f75f43e1bc5b (at 10.8.27.23@o2ib6) reconnecting Jul 11 20:10:43 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 20:12:27 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:14:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 20:14:15 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 11 20:15:06 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 11 20:15:06 fir-md1-s1 kernel: Lustre: 20571:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 34215 previous similar messages Jul 11 20:15:42 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:15:42 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Jul 11 20:15:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 20:15:48 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 11 20:18:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 20:18:53 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 11 20:20:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 20:20:44 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 11 20:22:27 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:22:27 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Jul 11 20:26:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 20:26:16 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 20:29:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 20:29:07 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 11 20:30:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 92201019-2a0e-37b3-944e-b91d23afff01 (at 10.8.17.26@o2ib6) reconnecting Jul 11 20:30:53 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 11 20:31:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 20:31:30 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 11 20:31:47 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:31:47 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Jul 11 20:36:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 20:36:16 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 11 20:39:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 20:39:19 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 11 20:41:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 20:41:00 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 20:44:20 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 20:44:20 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages Jul 11 20:46:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 20:46:29 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 11 20:46:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 20:46:30 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 11 20:49:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 20:49:32 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 11 20:51:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b4dc4310-abd3-57a8-960f-a27b33e667d3 (at 10.8.27.7@o2ib6) reconnecting Jul 11 20:51:04 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 20:57:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 20:57:16 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 20:57:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 20:57:48 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 11 20:59:38 fir-md1-s1 kernel: Lustre: 21411:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562903971/real 1562903971] req@ffff8f109fecec00 x1636730701721232/t0(0) o106->fir-MDT0002@10.9.106.54@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562903978 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 20:59:38 fir-md1-s1 kernel: Lustre: 21411:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 11 20:59:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 20:59:42 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 11 20:59:46 fir-md1-s1 kernel: Lustre: 23576:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0fb5fe4500 x1637050144562656/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:21/0 lens 480/568 e 1 to 0 dl 1562903991 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 20:59:59 fir-md1-s1 kernel: Lustre: 20571:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562903992/real 1562903992] req@ffff8f0bbd201200 x1636730701721200/t0(0) o106->fir-MDT0002@10.9.106.54@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562903999 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 11 20:59:59 fir-md1-s1 kernel: Lustre: 20571:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 11 21:00:12 fir-md1-s1 kernel: Lustre: 21452:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f2e868b0c00 x1638729918041728/t0(0) o101->957c1ad0-d547-b44d-0f14-5f92c3213a3d@10.8.15.3@o2ib6:17/0 lens 1800/3288 e 0 to 0 dl 1562904017 ref 2 fl Interpret:/0/0 rc 0/0 Jul 11 21:00:12 fir-md1-s1 kernel: Lustre: 21452:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 11 21:00:22 fir-md1-s1 kernel: LustreError: 23716:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.106.54@o2ib4) failed to reply to blocking AST (req@ffff8f2661a6ec00 x1636730702142368 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f24f8a82400/0x5d9ee63b0f89a974 lrc: 4/0,0 mode: PR/PR res: [0x2c002c32e:0x413:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.106.54@o2ib4 remote: 0x1fbe402371cceb58 expref: 1619 pid: 97644 timeout: 2019104 lvb_type: 0 Jul 11 21:00:22 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.9.106.54@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 11 21:00:22 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.9.106.54@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8f24f8a82400/0x5d9ee63b0f89a974 lrc: 4/0,0 mode: PR/PR res: [0x2c002c32e:0x413:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.106.54@o2ib4 remote: 0x1fbe402371cceb58 expref: 1620 pid: 97644 timeout: 0 lvb_type: 0 Jul 11 21:00:22 fir-md1-s1 kernel: Lustre: 21411:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:31s); client may timeout. req@ffff8f0f37cd7b00 x1637050144562816/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:21/0 lens 480/536 e 1 to 0 dl 1562903991 ref 1 fl Complete:/0/0 rc 301/301 Jul 11 21:00:22 fir-md1-s1 kernel: Lustre: 21411:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jul 11 21:01:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 21:01:13 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 11 21:03:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d101a6e2-e864-769d-b612-f06b470f1e70 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f22bdf56800, cur 1562904186 expire 1562904036 last 1562903959 Jul 11 21:08:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 21:08:06 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 11 21:09:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 21:09:01 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 21:10:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 21:10:19 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 11 21:11:04 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 21:11:04 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Jul 11 21:11:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 21:11:15 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 21:12:16 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:12:46 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:12:52 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:12:52 fir-md1-s1 kernel: LustreError: 22650:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 11 21:12:57 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:12:57 fir-md1-s1 kernel: LustreError: 20506:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 11 21:13:02 fir-md1-s1 kernel: LustreError: 21740:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:13:02 fir-md1-s1 kernel: LustreError: 21740:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 11 21:13:12 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:13:12 fir-md1-s1 kernel: LustreError: 46557:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 11 21:13:32 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:13:32 fir-md1-s1 kernel: LustreError: 46523:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 14 previous similar messages Jul 11 21:14:07 fir-md1-s1 kernel: LustreError: 21740:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:14:07 fir-md1-s1 kernel: LustreError: 21740:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22 previous similar messages Jul 11 21:18:29 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:18:29 fir-md1-s1 kernel: LustreError: 46538:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 54 previous similar messages Jul 11 21:19:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 21:21:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 21:21:00 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 21:21:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 21:21:04 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 11 21:21:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 21:21:52 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 11 21:31:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 21:31:09 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 11 21:31:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 21:31:25 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 11 21:32:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 21:32:05 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 21:36:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 21:36:01 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 21:41:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 21:41:23 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 11 21:41:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 21:41:29 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 11 21:42:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 21:42:13 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 21:50:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 21:52:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 21:52:34 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 11 21:52:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 21:52:34 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 11 21:52:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 21:52:43 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 11 21:54:48 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:54:48 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 3 previous similar messages Jul 11 21:55:25 fir-md1-s1 kernel: LustreError: 21793:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:55:59 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:55:59 fir-md1-s1 kernel: LustreError: 21709:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 1 previous similar message Jul 11 21:57:32 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 21:57:32 fir-md1-s1 kernel: LustreError: 22431:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 2 previous similar messages Jul 11 22:00:05 fir-md1-s1 kernel: LustreError: 23093:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 32768 GRANT, real grant 0 Jul 11 22:00:05 fir-md1-s1 kernel: LustreError: 23093:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 5 previous similar messages Jul 11 22:01:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9d8e287a-76f1-2fbc-54c1-19b634c62b63 (at 10.8.24.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24ee4acc00, cur 1562907687 expire 1562907537 last 1562907460 Jul 11 22:01:27 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 11 22:02:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 22:02:40 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 11 22:02:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 22:02:40 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 22:02:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 22:02:47 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 11 22:03:29 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 11 22:03:29 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Jul 11 22:05:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 22:05:18 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 11 22:12:36 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 22:12:36 fir-md1-s1 kernel: LustreError: 21682:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 12 previous similar messages Jul 11 22:12:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 22:12:46 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 11 22:13:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 22:13:00 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 11 22:13:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 22:13:54 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 11 22:18:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 22:21:37 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 22:21:37 fir-md1-s1 kernel: LustreError: 44034:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 17 previous similar messages Jul 11 22:23:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 22:23:09 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 22:23:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 22:23:09 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 11 22:24:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 22:24:10 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 11 22:26:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f30c5ff7c00, cur 1562909186 expire 1562909036 last 1562908959 Jul 11 22:26:26 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 11 22:29:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 22:31:43 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 22:31:43 fir-md1-s1 kernel: LustreError: 21715:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22 previous similar messages Jul 11 22:33:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 22:33:11 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 11 22:33:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 11 22:33:11 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 11 22:34:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 22:34:18 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 11 22:42:03 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 32768 GRANT, real grant 0 Jul 11 22:42:03 fir-md1-s1 kernel: LustreError: 21364:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 11 22:43:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 22:43:18 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 11 22:43:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 22:43:33 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 22:44:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 22:44:37 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 11 22:52:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 65863bd2-5bf4-3857-2c85-73178bef5ac4 (at 10.9.103.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34f4040c00, cur 1562910731 expire 1562910581 last 1562910504 Jul 11 22:52:20 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 22:52:20 fir-md1-s1 kernel: LustreError: 46512:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 11 22:52:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 22:52:40 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 11 22:53:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 639ee15e-2da6-9d93-315b-2c6ce5340bd5 (at 10.8.26.2@o2ib6) Jul 11 22:53:18 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 11 22:53:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 22:53:42 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 11 22:54:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 22:54:45 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 11 22:57:56 fir-md1-s1 kernel: Lustre: 23741:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562911069/real 1562911069] req@ffff8f30d62d4e00 x1636730844511680/t0(0) o106->fir-MDT0000@10.8.11.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562911076 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 11 22:57:56 fir-md1-s1 kernel: Lustre: 23741:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 11 23:02:32 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 23:02:32 fir-md1-s1 kernel: LustreError: 22432:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22 previous similar messages Jul 11 23:03:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 11 23:03:26 fir-md1-s1 kernel: Lustre: Skipped 163845 previous similar messages Jul 11 23:03:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 11 23:03:59 fir-md1-s1 kernel: Lustre: Skipped 163803 previous similar messages Jul 11 23:04:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 23:04:53 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 11 23:13:04 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 23:13:04 fir-md1-s1 kernel: LustreError: 20508:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 11 23:13:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 23:13:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 23:13:45 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 11 23:14:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 23:14:15 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 23:15:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 23:15:02 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 11 23:21:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 23:23:52 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 23:23:52 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 24 previous similar messages Jul 11 23:24:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 11 23:24:02 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 11 23:24:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 11 23:24:26 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 11 23:25:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 23:25:09 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 11 23:32:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 23:34:10 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 23:34:10 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 20 previous similar messages Jul 11 23:34:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 23:34:30 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 11 23:34:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 23:34:30 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 11 23:34:42 fir-md1-s1 kernel: Lustre: 10588:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 11 23:35:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 23:35:10 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 11 23:44:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 11 23:44:37 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 11 23:44:40 fir-md1-s1 kernel: LustreError: 22059:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 32768 GRANT, real grant 0 Jul 11 23:44:40 fir-md1-s1 kernel: LustreError: 22059:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 11 23:44:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 11 23:44:49 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 11 23:45:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 11 23:45:16 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 11 23:46:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 11 23:54:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 11 23:54:48 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 11 23:54:58 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 11 23:54:58 fir-md1-s1 kernel: LustreError: 46514:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 11 23:55:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 11 23:55:01 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 11 23:55:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 11 23:55:21 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 11 23:59:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:01:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:03:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:04:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 00:04:51 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 12 00:05:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 00:05:05 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 00:05:06 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 12 00:05:06 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 12 00:05:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 00:05:30 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 12 00:10:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:11:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:11:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:13:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:15:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 00:15:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 00:15:13 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 12 00:15:13 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 00:15:24 fir-md1-s1 kernel: LustreError: 46516:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 12 00:15:24 fir-md1-s1 kernel: LustreError: 46516:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 12 00:15:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 00:15:33 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 12 00:18:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:19:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f31dcb3e800, cur 1562915977 expire 1562915827 last 1562915750 Jul 12 00:19:37 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 00:20:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:25:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 00:25:20 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 12 00:25:32 fir-md1-s1 kernel: LustreError: 46560:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 12 00:25:32 fir-md1-s1 kernel: LustreError: 46560:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 21 previous similar messages Jul 12 00:25:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 00:25:34 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 12 00:25:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 00:25:35 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 00:33:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:35:44 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 32768 GRANT, real grant 0 Jul 12 00:35:44 fir-md1-s1 kernel: LustreError: 46584:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 12 00:35:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 12 00:35:48 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 12 00:35:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 00:35:48 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 12 00:36:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 00:36:07 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 12 00:37:04 fir-md1-s1 kernel: Lustre: 23689:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 12 00:41:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f231aefd000, cur 1562917303 expire 1562917153 last 1562917076 Jul 12 00:42:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:43:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:45:51 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 12 00:45:51 fir-md1-s1 kernel: LustreError: 46559:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 22 previous similar messages Jul 12 00:46:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 00:46:30 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 12 00:46:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 00:46:30 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 12 00:48:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 00:48:20 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 12 00:50:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 00:56:14 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 12 00:56:14 fir-md1-s1 kernel: LustreError: 25631:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 12 00:57:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 00:57:53 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 12 00:57:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 00:57:53 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 12 00:59:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 00:59:39 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 01:00:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 01:06:22 fir-md1-s1 kernel: LustreError: 23093:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 12 01:06:22 fir-md1-s1 kernel: LustreError: 23093:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 12 01:07:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 01:07:24 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 01:08:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 01:08:12 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 01:08:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 01:08:12 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 12 01:10:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 01:10:29 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 12 01:16:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 01:16:39 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) fir-MDT0002: cli ec935c16-6a63-f875-145b-2db5feba3892 claims 28672 GRANT, real grant 0 Jul 12 01:16:39 fir-md1-s1 kernel: LustreError: 21735:0:(tgt_grant.c:750:tgt_grant_check()) Skipped 23 previous similar messages Jul 12 01:18:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 01:18:25 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 01:18:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 01:18:25 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 12 01:20:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 01:20:48 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 12 01:29:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 01:29:01 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 12 01:29:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 01:29:48 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 12 01:30:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 01:30:52 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 01:30:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 01:30:54 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 12 01:39:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 01:39:29 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 12 01:40:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 01:40:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 01:40:58 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 12 01:40:58 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 01:41:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 01:41:26 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 12 01:49:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 01:49:35 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 12 01:51:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 01:51:03 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 01:52:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 01:52:24 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 01:59:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 01:59:13 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 01:59:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 01:59:36 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 12 02:01:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 02:01:27 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 02:03:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 02:03:27 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 12 02:09:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 02:09:39 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 12 02:11:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 02:11:40 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 12 02:12:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 02:12:23 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 02:14:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 02:14:39 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 12 02:19:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 02:19:45 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 12 02:21:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 12 02:21:45 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 02:23:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 02:23:42 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 02:25:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 02:25:21 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 12 02:30:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 02:30:29 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 12 02:31:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f0fb7d800, cur 1562923893 expire 1562923743 last 1562923666 Jul 12 02:31:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 02:31:56 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 02:34:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 02:34:50 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 12 02:35:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 02:35:21 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 12 02:40:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 02:40:34 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 12 02:43:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 02:43:38 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 02:45:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 02:45:32 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 02:46:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 02:46:39 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 02:50:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 02:50:40 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 12 02:54:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 02:54:15 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 12 02:56:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 02:56:30 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 02:58:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 02:58:53 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 03:01:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 03:01:02 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 12 03:04:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 03:04:35 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 03:06:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 03:06:36 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 12 03:11:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 03:11:12 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 12 03:14:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 03:14:57 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 03:17:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 03:17:17 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 03:19:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 03:19:33 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 03:21:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 03:21:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 03:21:14 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 03:21:14 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 12 03:24:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 03:24:58 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 12 03:25:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 03:25:14 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 03:27:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 03:27:19 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 12 03:31:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 03:31:21 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 03:31:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 03:31:26 fir-md1-s1 kernel: Lustre: Skipped 106 previous similar messages Jul 12 03:35:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 03:35:49 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 03:41:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 03:41:05 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 12 03:41:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 03:41:54 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 03:44:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 03:44:20 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 03:45:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 03:45:56 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 12 03:51:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 03:51:15 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 03:51:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 03:51:56 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 12 03:54:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 03:54:36 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 03:55:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 03:55:57 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 04:02:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 04:02:13 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 12 04:03:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 12 04:03:06 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 12 04:07:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 04:07:41 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 12 04:08:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 04:09:29 fir-md1-s1 kernel: Lustre: 23649:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929762/real 1562929762] req@ffff8f07bf2e1800 x1636731016960144/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929769 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 12 04:09:29 fir-md1-s1 kernel: Lustre: 23649:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 12 04:09:36 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929769/real 1562929769] req@ffff8f1533c1aa00 x1636731016960160/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929776 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 04:09:37 fir-md1-s1 kernel: Lustre: 23591:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f148c86bc00 x1637052831020032/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:12/0 lens 480/568 e 1 to 0 dl 1562929782 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 04:09:43 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929776/real 1562929776] req@ffff8f1533c1aa00 x1636731016960160/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929783 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 04:09:43 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 12 04:09:50 fir-md1-s1 kernel: Lustre: 23649:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929783/real 1562929783] req@ffff8f07bf2e1800 x1636731016960144/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929790 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 04:09:50 fir-md1-s1 kernel: Lustre: 23649:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 12 04:09:57 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929790/real 1562929790] req@ffff8f1533c1aa00 x1636731016960160/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929797 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 04:09:57 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 12 04:10:11 fir-md1-s1 kernel: Lustre: 23649:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929804/real 1562929804] req@ffff8f07bf2e1800 x1636731016960144/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929811 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 04:10:11 fir-md1-s1 kernel: Lustre: 23649:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 12 04:10:32 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929825/real 1562929825] req@ffff8f1533c1aa00 x1636731016960160/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929832 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 04:10:32 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 12 04:11:14 fir-md1-s1 kernel: Lustre: 23649:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929867/real 1562929867] req@ffff8f07bf2e1800 x1636731016960144/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929874 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 04:11:14 fir-md1-s1 kernel: Lustre: 23649:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 12 04:12:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 04:12:28 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 12 04:12:31 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562929944/real 1562929944] req@ffff8f1533c1aa00 x1636731016960160/t0(0) o106->fir-MDT0002@10.9.103.1@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562929951 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 04:12:31 fir-md1-s1 kernel: Lustre: 25681:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 12 04:12:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d5f6cf15-2331-01d2-988c-2d20adf007a2 (at 10.9.103.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2522e51800, cur 1562929959 expire 1562929809 last 1562929732 Jul 12 04:12:39 fir-md1-s1 kernel: Lustre: 23649:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (83:114s); client may timeout. req@ffff8f148c86bc00 x1637052831020032/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:12/0 lens 480/536 e 1 to 0 dl 1562929845 ref 1 fl Complete:/0/0 rc 301/301 Jul 12 04:12:39 fir-md1-s1 kernel: Lustre: 23649:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jul 12 04:14:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 04:14:38 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 12 04:17:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 04:17:53 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 12 04:22:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 04:22:29 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 12 04:24:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 04:24:33 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 12 04:24:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 04:24:41 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 12 04:28:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 04:28:08 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 12 04:32:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 04:32:33 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 12 04:34:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 04:34:47 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 12 04:38:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 04:38:15 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 04:38:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 04:38:49 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 12 04:43:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 04:43:05 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 12 04:45:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 04:45:56 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 12 04:48:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 12 04:48:18 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 04:53:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 04:53:09 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 12 04:56:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 04:56:40 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 12 04:57:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 04:57:26 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 04:58:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 04:58:37 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 12 05:03:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 05:03:40 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 12 05:07:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 05:07:23 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 05:08:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 05:08:42 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 12 05:11:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 05:11:17 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 05:13:41 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 05:13:41 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 12 05:18:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 05:18:39 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 12 05:19:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 05:19:42 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 12 05:23:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 05:23:46 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 12 05:29:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 05:29:00 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 12 05:30:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 05:30:03 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 05:33:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 05:33:47 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 12 05:39:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 05:39:41 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 12 05:40:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 05:40:10 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 12 05:40:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 05:40:40 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 12 05:43:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 05:43:58 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 12 05:44:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 05:49:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 05:50:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 05:50:19 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 05:50:57 fir-md1-s1 kernel: Lustre: 21312:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562935850/real 1562935850] req@ffff8f0dbc6cbf00 x1636731058646896/t0(0) o106->fir-MDT0002@10.9.103.22@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562935857 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 12 05:50:57 fir-md1-s1 kernel: Lustre: 21312:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 12 05:51:05 fir-md1-s1 kernel: Lustre: 23605:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0736e85d00 x1637053330878608/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:10/0 lens 480/568 e 1 to 0 dl 1562935870 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 05:51:05 fir-md1-s1 kernel: Lustre: 23605:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 12 05:51:18 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562935871/real 1562935871] req@ffff8f073eb56c00 x1636731058646928/t0(0) o106->fir-MDT0002@10.9.103.22@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562935878 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 05:51:18 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 12 05:52:00 fir-md1-s1 kernel: Lustre: 23691:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562935913/real 1562935913] req@ffff8f073eb56900 x1636731058646912/t0(0) o106->fir-MDT0002@10.9.103.22@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1562935920 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 05:52:00 fir-md1-s1 kernel: Lustre: 23691:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 12 05:52:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 12 05:52:08 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 12 05:53:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bd387039-9e5c-0c5b-0227-8087faaf7a40 (at 10.9.103.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4536596000, cur 1562935988 expire 1562935838 last 1562935761 Jul 12 05:53:08 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 12 05:55:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 05:55:03 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 12 06:01:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 06:01:07 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 06:03:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 06:03:22 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 06:05:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 06:05:05 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 12 06:11:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 06:11:16 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 12 06:11:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 06:11:47 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 06:13:59 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 06:13:59 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 12 06:15:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 06:15:06 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 06:16:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 06:22:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 06:22:27 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 12 06:24:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 06:24:55 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 12 06:25:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 06:25:06 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 12 06:33:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 06:33:06 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 12 06:34:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 06:34:56 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 06:35:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 06:35:06 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 06:37:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 06:41:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 06:42:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 06:43:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 06:43:18 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 12 06:45:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 06:45:20 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 12 06:47:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 06:47:15 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 12 06:53:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 06:53:25 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 12 06:54:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 06:55:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 06:55:21 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jul 12 06:58:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 06:58:32 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 12 07:00:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:02:39 fir-md1-s1 kernel: Lustre: 23563:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 12 07:03:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 07:03:27 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 12 07:04:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:05:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 07:05:27 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 12 07:05:29 fir-md1-s1 kernel: Lustre: 23591:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 Jul 12 07:09:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 07:09:12 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 07:11:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:13:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 07:13:54 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 12 07:15:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 07:15:45 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 07:16:08 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f37a4d77000, cur 1562940968 expire 1562940818 last 1562940741 Jul 12 07:16:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 07:20:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 07:20:04 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 07:24:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 07:24:06 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 12 07:26:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 07:26:02 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 12 07:26:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:27:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:28:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:29:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:30:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:32:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:32:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 07:32:27 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 12 07:34:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 07:34:14 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 07:36:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 07:36:43 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 12 07:39:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:39:40 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 07:42:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 07:42:31 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 07:44:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 07:44:23 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 07:44:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:44:48 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 07:46:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 07:46:48 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 12 07:54:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 07:54:25 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 12 07:55:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 07:55:27 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 12 07:56:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 07:56:19 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 12 07:57:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 07:57:05 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 12 08:04:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 08:04:41 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 08:07:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 08:07:27 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 12 08:07:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 08:07:31 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 12 08:14:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 08:14:50 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 08:16:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 08:16:56 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 12 08:17:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 08:17:33 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 12 08:17:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 08:17:33 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 12 08:17:56 fir-md1-s1 kernel: Lustre: 23607:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562944669/real 1562944669] req@ffff8f2ec4477200 x1636731118387008/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562944676 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 12 08:17:56 fir-md1-s1 kernel: Lustre: 23607:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 27 previous similar messages Jul 12 08:18:04 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0e22d5aa00 x1633658408530896/t0(0) o36->60a9f157-4802-e53d-dccf-19f0d690f2d1@10.9.0.1@o2ib4:9/0 lens 496/448 e 1 to 0 dl 1562944689 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 08:18:04 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 161 previous similar messages Jul 12 08:18:05 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2ca9e7f500 x1631702352577056/t0(0) o101->2d384d58-fd4c-f6d6-342b-6f9f296484e1@10.9.101.46@o2ib4:10/0 lens 1768/0 e 1 to 0 dl 1562944690 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 12 08:18:05 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 348 previous similar messages Jul 12 08:18:06 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2e89247500 x1631636431594032/t0(0) o101->0d8fe43d-85f9-8061-e5fc-2e0ec8fbd940@10.8.7.11@o2ib6:11/0 lens 576/0 e 1 to 0 dl 1562944691 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 12 08:18:06 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 212 previous similar messages Jul 12 08:18:08 fir-md1-s1 kernel: Lustre: 23588:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3239eaf800 x1638092360824656/t0(0) o101->95c23571-6ded-28b5-8b2e-63d85e709c23@10.8.15.4@o2ib6:13/0 lens 1768/0 e 1 to 0 dl 1562944693 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 12 08:18:08 fir-md1-s1 kernel: Lustre: 23588:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 682 previous similar messages Jul 12 08:18:10 fir-md1-s1 kernel: Lustre: 23607:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562944683/real 1562944683] req@ffff8f2ec4477200 x1636731118387008/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562944690 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 08:18:10 fir-md1-s1 kernel: Lustre: 23607:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 12 08:18:12 fir-md1-s1 kernel: Lustre: 23588:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f27c342fb00 x1636577844268464/t0(0) o101->42f49237-eaa5-3549-e9cf-6b0ef8d87e1a@10.9.112.7@o2ib4:17/0 lens 576/0 e 1 to 0 dl 1562944697 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 12 08:18:12 fir-md1-s1 kernel: Lustre: 23588:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 296 previous similar messages Jul 12 08:18:20 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2993ae6000 x1638080775806352/t0(0) o101->c0496bb5-bb8d-8fb8-13d2-918f029a4d08@10.8.26.34@o2ib6:25/0 lens 576/0 e 1 to 0 dl 1562944705 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 12 08:18:20 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 453 previous similar messages Jul 12 08:18:24 fir-md1-s1 kernel: LustreError: 23607:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.7@o2ib6) failed to reply to blocking AST (req@ffff8f2ec3c80f00 x1636731118397200 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f348cc0cc80/0x5d9ee63a22157135 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 882 type: IBT flags: 0x60200400000020 nid: 10.8.27.7@o2ib6 remote: 0x99e7546245a2f947 expref: 375 pid: 23741 timeout: 2059786 lvb_type: 0 Jul 12 08:18:24 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.27.7@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 12 08:18:24 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.27.7@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f348cc0cc80/0x5d9ee63a22157135 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 882 type: IBT flags: 0x60200400000020 nid: 10.8.27.7@o2ib6 remote: 0x99e7546245a2f947 expref: 376 pid: 23741 timeout: 0 lvb_type: 0 Jul 12 08:18:31 fir-md1-s1 kernel: Lustre: 23607:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562944704/real 1562944704] req@ffff8f2ec4477200 x1636731118387008/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562944711 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 08:18:31 fir-md1-s1 kernel: Lustre: 23607:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 12 08:18:36 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2d99175a00 x1638731879984176/t0(0) o101->159ddaf1-ce95-3830-127f-4856eec7f12f@10.9.116.1@o2ib4:11/0 lens 576/0 e 0 to 0 dl 1562944721 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 12 08:18:36 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2456 previous similar messages Jul 12 08:19:08 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f32346bc800 x1631556903864304/t0(0) o101->2faef2d8-dc67-f384-07b6-111f344194c1@10.9.101.65@o2ib4:13/0 lens 576/0 e 0 to 0 dl 1562944753 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 12 08:19:08 fir-md1-s1 kernel: Lustre: 23676:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3432 previous similar messages Jul 12 08:19:13 fir-md1-s1 kernel: Lustre: 23607:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562944746/real 1562944746] req@ffff8f2ec4477200 x1636731118387008/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562944753 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 08:19:13 fir-md1-s1 kernel: Lustre: 23607:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 12 08:19:19 fir-md1-s1 kernel: LustreError: 97646:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562944669, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f236c869200/0x5d9ee63ba86a12ae lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97646 timeout: 0 lvb_type: 0 Jul 12 08:19:19 fir-md1-s1 kernel: LustreError: 97646:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 7 previous similar messages Jul 12 08:19:20 fir-md1-s1 kernel: LustreError: 23699:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562944670, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f3c80b5f2c0/0x5d9ee63ba86a58e6 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23699 timeout: 0 lvb_type: 0 Jul 12 08:19:20 fir-md1-s1 kernel: LustreError: 23699:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 238 previous similar messages Jul 12 08:19:22 fir-md1-s1 kernel: LustreError: 10196:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562944672, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f13b7a4bcc0/0x5d9ee63ba86ac948 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10196 timeout: 0 lvb_type: 0 Jul 12 08:19:22 fir-md1-s1 kernel: LustreError: 10196:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 64 previous similar messages Jul 12 08:19:26 fir-md1-s1 kernel: LustreError: 21420:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562944676, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f05083c4ec0/0x5d9ee63ba86b9a6f lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21420 timeout: 0 lvb_type: 0 Jul 12 08:19:26 fir-md1-s1 kernel: LustreError: 21420:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 105 previous similar messages Jul 12 08:20:12 fir-md1-s1 kernel: Lustre: 23588:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f263f9cc200 x1631563254427968/t0(0) o101->1b1a33fc-473d-0f6c-9f25-a44e13708af4@10.8.8.3@o2ib6:17/0 lens 576/0 e 0 to 0 dl 1562944817 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 12 08:20:12 fir-md1-s1 kernel: Lustre: 23588:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 105533 previous similar messages Jul 12 08:20:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 9c359101-0ed1-475a-f1ab-59f22d57209c (at 10.8.22.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34ed7a1800, cur 1562944821 expire 1562944671 last 1562944594 Jul 12 08:20:22 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 5990ca21-7371-f423-fd1a-20751dbd1238 (at 10.9.103.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f15360d7000, cur 1562944822 expire 1562944672 last 1562944595 Jul 12 08:20:22 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 12 08:20:22 fir-md1-s1 kernel: Lustre: 23589:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:115s); client may timeout. req@ffff8f090e628f00 x1637053928455488/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:27/0 lens 616/0 e 0 to 0 dl 1562944707 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 12 08:20:22 fir-md1-s1 kernel: LustreError: 23736:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.104.22@o2ib4: deadline 30:1s ago req@ffff8f3e90db2700 x1631571371738656/t0(0) o101->c1d9f0f7-d490-e556-ed11-756e6b122018@10.9.104.22@o2ib4:21/0 lens 576/0 e 0 to 0 dl 1562944821 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jul 12 08:20:22 fir-md1-s1 kernel: LustreError: 23736:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 5 previous similar messages Jul 12 08:20:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 08:20:22 fir-md1-s1 kernel: Lustre: 23589:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 93557 previous similar messages Jul 12 08:20:22 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 12 08:24:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 08:24:52 fir-md1-s1 kernel: Lustre: Skipped 20771 previous similar messages Jul 12 08:26:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client a68d9e6e-5a78-ad1d-4abc-986d23128d99 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1ae3559000, cur 1562945193 expire 1562945043 last 1562944966 Jul 12 08:26:33 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 08:28:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 08:28:36 fir-md1-s1 kernel: Lustre: Skipped 20806 previous similar messages Jul 12 08:28:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 08:28:49 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 08:30:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 08:30:50 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 08:33:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 66aa3c20-db51-1ee2-67da-24de875c7f64 (at 10.9.113.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2a79848000, cur 1562945628 expire 1562945478 last 1562945401 Jul 12 08:33:48 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 08:34:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 08:34:54 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 12 08:36:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 08:36:47 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 08:38:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 08:38:42 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 12 08:38:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 12 08:38:52 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 12 08:40:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 4ef54d74-26c7-dd87-45f7-921d9e4ba654 (at 10.9.115.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1e96392400, cur 1562946046 expire 1562945896 last 1562945819 Jul 12 08:40:46 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 08:45:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 08:45:06 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 08:47:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client c052722f-fb7b-9d40-a2d8-d22451dc2117 (at 10.9.115.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f22a6103000, cur 1562946471 expire 1562946321 last 1562946244 Jul 12 08:47:51 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 08:48:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 2b7395d9-c40c-f531-147e-33ca0a08dcda (at 10.8.22.12@o2ib6) Jul 12 08:48:46 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 12 08:48:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 08:48:49 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 08:49:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 08:49:17 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 12 08:55:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 08:55:32 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 08:58:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 08:58:50 fir-md1-s1 kernel: Lustre: Skipped 136 previous similar messages Jul 12 08:59:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 12 08:59:22 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 09:01:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 09:01:10 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 09:05:22 fir-md1-s1 kernel: Lustre: 25676:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562947511/real 1562947511] req@ffff8f1299d1a100 x1636731136281312/t0(0) o104->fir-MDT0000@10.9.115.10@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562947522 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 12 09:05:22 fir-md1-s1 kernel: Lustre: 25676:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 12 09:05:26 fir-md1-s1 kernel: Lustre: 20723:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1c5e9f3600 x1631559599111984/t0(0) o101->bb17aca1-57d8-f36a-a79b-bcdcd36ec002@10.8.18.20@o2ib6:1/0 lens 576/3264 e 1 to 0 dl 1562947531 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 09:05:26 fir-md1-s1 kernel: Lustre: 20723:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3630 previous similar messages Jul 12 09:05:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bb17aca1-57d8-f36a-a79b-bcdcd36ec002 (at 10.8.18.20@o2ib6) reconnecting Jul 12 09:05:32 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 09:05:33 fir-md1-s1 kernel: Lustre: 25676:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562947522/real 1562947522] req@ffff8f1299d1a100 x1636731136281312/t0(0) o104->fir-MDT0000@10.9.115.10@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562947533 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 09:05:42 fir-md1-s1 kernel: Lustre: 20723:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f23edf04800 x1634116848491408/t0(0) o101->2aa758e4-fe35-42c9-321f-e6d541fd5bfd@10.8.27.17@o2ib6:17/0 lens 576/0 e 1 to 0 dl 1562947547 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 12 09:05:42 fir-md1-s1 kernel: Lustre: 20723:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2566 previous similar messages Jul 12 09:05:55 fir-md1-s1 kernel: Lustre: 25676:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562947544/real 1562947544] req@ffff8f1299d1a100 x1636731136281312/t0(0) o104->fir-MDT0000@10.9.115.10@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562947555 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 09:05:55 fir-md1-s1 kernel: Lustre: 25676:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 12 09:06:14 fir-md1-s1 kernel: Lustre: 20464:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f319fcfdd00 x1631567667024064/t0(0) o101->2dd7454a-4666-cb77-2a9b-10ada81c5a76@10.8.18.27@o2ib6:19/0 lens 576/0 e 0 to 0 dl 1562947579 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 12 09:06:14 fir-md1-s1 kernel: Lustre: 20464:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 40577 previous similar messages Jul 12 09:06:39 fir-md1-s1 kernel: Lustre: 25676:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562947588/real 1562947588] req@ffff8f1299d1a100 x1636731136281312/t0(0) o104->fir-MDT0000@10.9.115.10@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1562947599 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 09:06:39 fir-md1-s1 kernel: Lustre: 25676:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 12 09:06:41 fir-md1-s1 kernel: LustreError: 20952:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562947511, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f15bdafee40/0x5d9ee63bad350cbb lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20952 timeout: 0 lvb_type: 0 Jul 12 09:06:41 fir-md1-s1 kernel: LustreError: 20952:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 24 previous similar messages Jul 12 09:06:42 fir-md1-s1 kernel: LustreError: 23648:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562947512, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f0b0d06d340/0x5d9ee63bad353b84 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23648 timeout: 0 lvb_type: 0 Jul 12 09:06:42 fir-md1-s1 kernel: LustreError: 23648:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 244 previous similar messages Jul 12 09:06:44 fir-md1-s1 kernel: LustreError: 21369:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562947514, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f205e2772c0/0x5d9ee63bad35416c lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21369 timeout: 0 lvb_type: 0 Jul 12 09:06:44 fir-md1-s1 kernel: LustreError: 21369:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 45 previous similar messages Jul 12 09:06:48 fir-md1-s1 kernel: LustreError: 21145:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562947518, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f3fa133b600/0x5d9ee63bad354c86 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21145 timeout: 0 lvb_type: 0 Jul 12 09:06:48 fir-md1-s1 kernel: LustreError: 21145:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 70 previous similar messages Jul 12 09:07:18 fir-md1-s1 kernel: Lustre: 23622:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2eea337500 x1631571215696624/t0(0) o101->3ef17f0c-d35b-8428-c1da-c84a40a8bdbc@10.9.101.71@o2ib4:23/0 lens 576/0 e 0 to 0 dl 1562947643 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 12 09:07:18 fir-md1-s1 kernel: Lustre: 23622:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 102499 previous similar messages Jul 12 09:07:45 fir-md1-s1 kernel: LustreError: 25676:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.115.10@o2ib4) failed to reply to blocking AST (req@ffff8f1299d1a100 x1636731136281312 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8f1b321e8000/0x5d9ee63baa823441 lrc: 4/0,0 mode: PR/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x60200400000020 nid: 10.9.115.10@o2ib4 remote: 0x9dc480fd8d5c83c4 expref: 12 pid: 20720 timeout: 2062863 lvb_type: 0 Jul 12 09:07:45 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.9.115.10@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Jul 12 09:07:45 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.9.115.10@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff8f1b321e8000/0x5d9ee63baa823441 lrc: 3/0,0 mode: PR/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x60200400000020 nid: 10.9.115.10@o2ib4 remote: 0x9dc480fd8d5c83c4 expref: 13 pid: 20720 timeout: 0 lvb_type: 0 Jul 12 09:07:45 fir-md1-s1 kernel: Lustre: 23730:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:123s); client may timeout. req@ffff8f4048976000 x1631541592774480/t0(0) o101->5735cd86-3a30-362c-bc05-c634d3fa1859@10.9.107.11@o2ib4:12/0 lens 576/0 e 1 to 0 dl 1562947542 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 12 09:07:45 fir-md1-s1 kernel: LustreError: 23637:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.115.4@o2ib4: deadline 100:22s ago req@ffff8f1d7818e600 x1638869204205952/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1562947643 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 12 09:07:45 fir-md1-s1 kernel: LustreError: 23637:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 77 previous similar messages Jul 12 09:07:45 fir-md1-s1 kernel: Lustre: 23730:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 92935 previous similar messages Jul 12 09:07:45 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 12 09:07:45 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Jul 12 09:08:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client dc3fcf2e-e5b5-2903-ff5f-2681ca61121a (at 10.9.115.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f39cbea9800, cur 1562947703 expire 1562947553 last 1562947476 Jul 12 09:08:23 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 09:08:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 09:08:56 fir-md1-s1 kernel: Lustre: Skipped 26772 previous similar messages Jul 12 09:09:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 09:09:24 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 12 09:15:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 09:15:33 fir-md1-s1 kernel: Lustre: Skipped 26731 previous similar messages Jul 12 09:19:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 09:19:01 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 09:19:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 09:19:03 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 12 09:20:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 09:20:54 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 12 09:26:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 09:26:42 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 09:29:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 09:29:13 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 12 09:29:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 09:29:30 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 12 09:30:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 09:30:54 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 12 09:36:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 09:36:45 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 12 09:39:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 09:39:22 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 12 09:40:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 09:40:48 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 12 09:41:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 09:41:16 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 09:47:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 12 09:47:17 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 09:49:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 09:49:30 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 12 09:51:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 09:51:30 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 09:51:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 09:51:56 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 09:57:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 09:57:32 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 12 09:58:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 507fe63e-2eba-dc91-49a6-94f7b912a620 (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4502cd5c00, cur 1562950709 expire 1562950559 last 1562950482 Jul 12 09:58:29 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 12 09:59:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 09:59:31 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 12 10:02:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 10:02:36 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 12 10:06:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 10:06:08 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 12 10:08:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 10:08:20 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 10:09:38 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 10:09:38 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 12 10:12:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 10:12:37 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 12 10:18:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 10:18:00 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 10:18:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 10:18:21 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 10:19:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 10:19:53 fir-md1-s1 kernel: Lustre: Skipped 97 previous similar messages Jul 12 10:22:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 10:22:40 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 10:28:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 10:28:27 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 12 10:30:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 10:30:04 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 12 10:33:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 10:33:58 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 10:38:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 10:38:35 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 12 10:40:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 10:40:09 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 12 10:41:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client d2e199c6-98a7-0717-8652-10bd2bf787b1 (at 10.8.24.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2530a40400, cur 1562953269 expire 1562953119 last 1562953042 Jul 12 10:41:09 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 10:44:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 10:44:00 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 12 10:48:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 10:48:49 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 10:49:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 10:50:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 10:50:22 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 12 10:54:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 10:56:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 10:56:06 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 12 10:58:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 10:58:52 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 11:00:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 11:00:01 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 11:00:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 11:00:27 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 11:06:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 11:06:19 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 11:08:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 11:08:11 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 11:08:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 2f8cf2ac-3786-7722-1124-7c8b6ba37f05 (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f23163fd400, cur 1562954899 expire 1562954749 last 1562954672 Jul 12 11:08:19 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 11:09:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 11:09:13 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 11:10:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 11:10:37 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 12 11:16:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 12 11:16:48 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 11:19:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 11:19:28 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 12 11:19:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 11:19:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 11:19:28 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 12 11:20:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 11:20:49 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 12 11:27:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 11:27:02 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 12 11:30:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 12 11:30:13 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 12 11:31:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 11:31:03 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 12 11:32:40 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562956353/real 1562956353] req@ffff8f10a7f30000 x1636731199047520/t0(0) o106->fir-MDT0000@10.8.27.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562956360 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 12 11:32:40 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 12 11:32:48 fir-md1-s1 kernel: Lustre: 23629:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0ad0ec1500 x1637054700457712/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:23/0 lens 480/568 e 1 to 0 dl 1562956373 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 11:32:48 fir-md1-s1 kernel: Lustre: 23629:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4553 previous similar messages Jul 12 11:32:54 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562956367/real 1562956367] req@ffff8f10a7f30000 x1636731199047520/t0(0) o106->fir-MDT0000@10.8.27.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562956374 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 11:32:54 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 12 11:33:15 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562956388/real 1562956388] req@ffff8f10a7f30000 x1636731199047520/t0(0) o106->fir-MDT0000@10.8.27.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562956395 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 11:33:15 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 12 11:33:57 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562956430/real 1562956430] req@ffff8f10a7f30000 x1636731199047520/t0(0) o106->fir-MDT0000@10.8.27.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562956437 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 11:33:57 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 12 11:35:14 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562956507/real 1562956507] req@ffff8f10a7f30000 x1636731199047520/t0(0) o106->fir-MDT0000@10.8.27.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562956514 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 11:35:14 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 12 11:35:51 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 2a234d0c-a8ab-8feb-25f1-bf6554cceb02 (at 10.9.106.57@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f153524c000, cur 1562956551 expire 1562956401 last 1562956324 Jul 12 11:35:51 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 11:35:53 fir-md1-s1 kernel: LNet: Service thread pid 23589 was inactive for 200.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 12 11:35:53 fir-md1-s1 kernel: Pid: 23589, comm: mdt00_071 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 12 11:35:53 fir-md1-s1 kernel: Call Trace: Jul 12 11:35:53 fir-md1-s1 kernel: [] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] mdt_do_glimpse+0x1e9/0x4c0 [mdt] Jul 12 11:35:53 fir-md1-s1 kernel: [] mdt_glimpse_enqueue+0x3d3/0x4f0 [mdt] Jul 12 11:35:53 fir-md1-s1 kernel: [] mdt_intent_glimpse+0x1f/0x30 [mdt] Jul 12 11:35:53 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 12 11:35:53 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 12 11:35:53 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 12 11:35:53 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 12 11:35:53 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 12 11:35:53 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1562956553.23589 Jul 12 11:35:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 88249aca-f8a5-51dd-af36-5041bca337b5 (at 10.8.16.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2522963400, cur 1562956557 expire 1562956407 last 1562956330 Jul 12 11:35:57 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 12 11:35:57 fir-md1-s1 kernel: LNet: Service thread pid 23589 completed after 203.98s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jul 12 11:35:57 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Jul 12 11:36:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 400d6bb2-cc30-d980-7d8b-e0cf4a3a30a0 (at 10.9.107.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f148c8f3800, cur 1562956561 expire 1562956411 last 1562956334 Jul 12 11:36:01 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 12 11:37:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 11:37:41 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 12 11:37:45 fir-md1-s1 kernel: Lustre: 21410:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f0cde604b00 x1637054711613040/t0(0) o101->6ee172d9-72a9-7fa2-230d-3850214207fa@10.0.10.3@o2ib7:20/0 lens 480/568 e 0 to 0 dl 1562956670 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 11:37:48 fir-md1-s1 kernel: Lustre: 10589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562956661/real 1562956661] req@ffff8f127ceb2d00 x1636731200892032/t0(0) o106->fir-MDT0002@10.8.27.35@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1562956668 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 11:37:48 fir-md1-s1 kernel: Lustre: 10589:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 12 11:40:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 11:40:14 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 11:40:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 11:40:16 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 12 11:40:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client be42b497-ab1b-8d58-3101-014aad577cfc (at 10.8.27.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f8ad6000, cur 1562956829 expire 1562956679 last 1562956602 Jul 12 11:40:29 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 12 11:41:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 11:41:04 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 12 11:41:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 11:41:36 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 11:44:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 11:44:42 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 11:48:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 11:48:38 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 11:50:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 11:50:38 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 11:51:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 11:51:06 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 12 11:51:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 11:51:32 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 11:53:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 62b59a8a-bc87-45e0-45ad-94363e33396b (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f330b675000, cur 1562957588 expire 1562957438 last 1562957361 Jul 12 11:53:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 11:58:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 11:58:53 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 12:01:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 12:01:11 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 12 12:01:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 12:01:42 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 12 12:02:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 12:02:07 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 12:10:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 12:10:20 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 12 12:11:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 19effcd6-8030-8ae1-d9d6-24266f7c8d3c (at 10.8.27.35@o2ib6) Jul 12 12:11:19 fir-md1-s1 kernel: Lustre: Skipped 107 previous similar messages Jul 12 12:11:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client dc10f947-e401-3136-94f5-752e472b9896 (at 10.9.103.10@o2ib4) in 207 seconds. I think it's dead, and I am evicting it. exp ffff8f1478a02400, cur 1562958693 expire 1562958543 last 1562958486 Jul 12 12:11:33 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 12:11:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 12:11:42 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 12:14:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 12:14:46 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 12 12:16:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 054d1548-cdcc-8b5b-1ec4-5ec77e76503f (at 10.8.12.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25013e4400, cur 1562959002 expire 1562958852 last 1562958775 Jul 12 12:16:42 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 12 12:21:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 12:21:14 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 12:21:32 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 12:21:32 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 12 12:21:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 12:21:54 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 12 12:23:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client dce47606-d438-ab63-01f6-1079880f0e28 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1a2a1bcc00, cur 1562959394 expire 1562959244 last 1562959167 Jul 12 12:23:14 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 12:29:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 12:29:20 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 12:31:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 84fd8c4b-6545-cd41-282d-ef5f651cba30 (at 10.8.17.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2529f98c00, cur 1562959916 expire 1562959766 last 1562959689 Jul 12 12:31:56 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 12:31:56 fir-md1-s1 kernel: LustreError: 55142:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f07828d8000 x1636731232215392/t0(0) o105->fir-MDT0002@10.8.17.11@o2ib6:15/16 lens 304/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 12 12:31:56 fir-md1-s1 kernel: LustreError: 55142:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jul 12 12:32:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 12:32:08 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 12:32:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 12:32:08 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 12 12:32:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 12:32:21 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 12 12:33:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 3a0d0e87-adb3-f22f-9e98-cf9d12330e59 (at 10.9.113.15@o2ib4) in 185 seconds. I think it's dead, and I am evicting it. exp ffff8f34f438e400, cur 1562959992 expire 1562959842 last 1562959807 Jul 12 12:33:12 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 12:33:54 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 8072542a-c77e-8c5c-c60e-0629def56e65 (at 10.9.113.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25208a8000, cur 1562960034 expire 1562959884 last 1562959807 Jul 12 12:42:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 12 12:42:31 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 12:42:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 12:42:31 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 12 12:42:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 12:42:56 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 12:53:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 12:53:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 12:53:02 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 12 12:53:02 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 12:53:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 12:53:11 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 12 12:58:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 12:58:07 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 13:03:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 12 13:03:02 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 12 13:03:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 13:03:02 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 12 13:03:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 13:03:10 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 13:04:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 13:04:05 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 12 13:05:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 13:05:55 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 13:13:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 13:13:05 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 12 13:13:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 13:13:05 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 12 13:15:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 4dc6ad45-c67c-15d0-5638-611b0defe5f9 (at 10.8.16.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f45018ea400, cur 1562962520 expire 1562962370 last 1562962293 Jul 12 13:15:20 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 12 13:15:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 13:15:37 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 12 13:18:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 6b363170-4e30-8684-0ee2-d3bef7a36f68 (at 10.9.103.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4518d83000, cur 1562962680 expire 1562962530 last 1562962453 Jul 12 13:18:00 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 13:18:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client abffd720-a2aa-412e-9038-98cd76f7763d (at 10.9.103.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fdc31c00, cur 1562962696 expire 1562962546 last 1562962469 Jul 12 13:18:16 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 12 13:19:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 13458280-a046-3a7f-2bec-0301aba013a1 (at 10.8.28.12@o2ib6) in 211 seconds. I think it's dead, and I am evicting it. exp ffff8f1473afa000, cur 1562962756 expire 1562962606 last 1562962545 Jul 12 13:19:16 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 13:19:32 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client d0d1dcda-abd5-29f1-1250-5971b6db7d8a (at 10.8.28.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34ea41b800, cur 1562962772 expire 1562962622 last 1562962545 Jul 12 13:19:32 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 13:21:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2104b6c000, cur 1562962869 expire 1562962719 last 1562962642 Jul 12 13:21:09 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 12 13:23:52 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 13:23:52 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 12 13:23:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 13:23:56 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 13:26:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 13:26:10 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 12 13:34:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 13:34:20 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 12 13:34:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 13:34:20 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 12 13:36:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 13:36:13 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 13:44:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 13:44:53 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 12 13:44:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 13:44:53 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 12 13:45:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 13:45:32 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 13:46:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 13:46:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 13:46:26 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 13:55:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 13:55:03 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 13:55:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 13:55:03 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 12 13:56:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 13:56:34 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 12 13:58:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 94b66082-9a8c-20cb-2dfb-0baa5381ec3e (at 10.9.104.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3504108400, cur 1562965126 expire 1562964976 last 1562964899 Jul 12 14:05:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 14:05:05 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 12 14:05:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 14:05:05 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 12 14:06:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 14:06:43 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 14:13:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 14:15:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 14:15:29 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 12 14:15:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 14:15:40 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 12 14:17:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 12 14:17:00 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 12 14:25:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 14:25:43 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 14:25:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 14:25:43 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 12 14:27:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 14:27:17 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 12 14:32:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 50f228d0-9830-8cb0-9089-89882ee52793 (at 10.9.113.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2528ac3000, cur 1562967139 expire 1562966989 last 1562966912 Jul 12 14:32:19 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 14:35:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 14:35:47 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 12 14:36:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 14:36:11 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 14:38:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 14:38:08 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 12 14:42:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 14:43:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 14:45:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 14:45:47 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 12 14:47:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 14:47:02 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 14:48:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 14:48:20 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 12 14:55:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 14:55:51 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 12 14:57:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 14:57:11 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 15:00:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 15:00:42 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 15:05:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:05:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 15:05:59 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 12 15:06:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:07:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 15:07:12 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 15:07:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:10:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 715f02cb-8e2e-f659-95b8-6785da84ae98 (at 10.8.30.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f269ebcf400, cur 1562969427 expire 1562969277 last 1562969200 Jul 12 15:10:27 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:12:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 15:12:30 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 12 15:13:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c271ddbf-2f8d-722d-f50f-1f7affd6178d (at 10.9.115.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0f057acc00, cur 1562969622 expire 1562969472 last 1562969395 Jul 12 15:13:42 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 12 15:16:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 8b6431ee-4a59-d217-cf47-d826ce17927f (at 10.9.112.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f363afd5000, cur 1562969776 expire 1562969626 last 1562969549 Jul 12 15:16:16 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:16:39 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 15:16:39 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 12 15:17:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 15:17:12 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 15:19:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:22:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:23:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 9c2eb81b-3f24-241f-5bf3-071355b5c7e1 (at 10.9.112.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d3673f000, cur 1562970211 expire 1562970061 last 1562969984 Jul 12 15:23:31 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:24:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:25:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 15:25:38 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 12 15:26:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 15:26:58 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 12 15:27:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 15:27:49 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 15:28:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:29:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:30:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:31:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:33:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 274acbe5-1f09-1bc7-1d04-06ba56c47198 (at 10.8.25.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d26bc5400, cur 1562970796 expire 1562970646 last 1562970569 Jul 12 15:33:16 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:35:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:37:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 15:37:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 15:37:00 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 12 15:37:00 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 12 15:38:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 15:38:03 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 15:38:40 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 12 15:40:42 fir-md1-s1 kernel: Lustre: 23565:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 12 15:40:57 fir-md1-s1 kernel: Lustre: 23706:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 12 15:40:57 fir-md1-s1 kernel: Lustre: 23706:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 32 previous similar messages Jul 12 15:41:05 fir-md1-s1 kernel: Lustre: 23687:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 12 15:41:08 fir-md1-s1 kernel: Lustre: 10589:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 12 15:42:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 96d3d94a-0025-4481-959e-9b59edd190d8 (at 10.9.113.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1c18054800, cur 1562971377 expire 1562971227 last 1562971150 Jul 12 15:42:57 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:44:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c623c6e5-2a28-10b9-ccff-a82c94121897 (at 10.8.15.6@o2ib6) in 201 seconds. I think it's dead, and I am evicting it. exp ffff8f29f7853000, cur 1562971453 expire 1562971303 last 1562971252 Jul 12 15:44:13 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:46:12 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 12 15:46:12 fir-md1-s1 kernel: Lustre: 21670:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 3 previous similar messages Jul 12 15:47:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 15:47:03 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 12 15:47:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 15:47:12 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 12 15:48:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 15:48:04 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 15:50:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 092321e4-f4f0-9526-3615-cc8623ccd65a (at 10.9.115.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f19774e1800, cur 1562971820 expire 1562971670 last 1562971593 Jul 12 15:50:20 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:50:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:51:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:52:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:53:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 83d92eeb-0189-3899-1ebe-4ec09cf09eb2 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f25df30c800, cur 1562972000 expire 1562971850 last 1562971773 Jul 12 15:53:20 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:54:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 15:57:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 15:57:16 fir-md1-s1 kernel: Lustre: Skipped 109 previous similar messages Jul 12 15:57:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 15:57:18 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 12 15:57:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 1461e82f-da19-d2c0-6023-1022ba7a9852 (at 10.9.115.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f191a006400, cur 1562972264 expire 1562972114 last 1562972037 Jul 12 15:57:44 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 15:58:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 15:58:06 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 16:00:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 16:03:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 16:07:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 16:07:20 fir-md1-s1 kernel: Lustre: Skipped 99 previous similar messages Jul 12 16:08:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 16:08:05 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 12 16:08:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 16:08:18 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 12 16:10:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5f76e786-77c9-ffeb-e686-315d04a0455d (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2e0f6eb400, cur 1562973056 expire 1562972906 last 1562972829 Jul 12 16:10:56 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 16:14:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 16:14:11 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 16:17:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 16:17:20 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 12 16:18:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 16:18:21 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 16:19:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 12 16:19:24 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 12 16:24:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 16:24:56 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 16:26:04 fir-md1-s1 kernel: Lustre: 50445:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562973957/real 1562973957] req@ffff8f1a499cfb00 x1636731502998496/t0(0) o104->fir-MDT0002@10.8.27.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562973964 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 12 16:26:04 fir-md1-s1 kernel: Lustre: 50445:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 44 previous similar messages Jul 12 16:26:12 fir-md1-s1 kernel: Lustre: 97669:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1bfea45400 x1638241009838816/t0(0) o101->b74b4b66-65f0-f951-331c-463b7f96e033@10.9.0.62@o2ib4:17/0 lens 1768/3288 e 1 to 0 dl 1562973977 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 16:26:12 fir-md1-s1 kernel: Lustre: 97669:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 12 16:26:17 fir-md1-s1 kernel: Lustre: 97656:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f22f5587500 x1638242439524656/t0(0) o101->83b4afa2-a367-a71c-8602-481ad43297ce@10.8.0.68@o2ib6:22/0 lens 592/3264 e 1 to 0 dl 1562973982 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 16:26:17 fir-md1-s1 kernel: Lustre: 97656:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 12 16:26:46 fir-md1-s1 kernel: Lustre: 50445:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1562973999/real 1562973999] req@ffff8f1a499cfb00 x1636731502998496/t0(0) o104->fir-MDT0002@10.8.27.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1562974006 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 12 16:26:46 fir-md1-s1 kernel: Lustre: 50445:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 12 16:26:52 fir-md1-s1 kernel: Lustre: 10585:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8f3366898c00 x1631578363371568/t0(0) o101->1135836c-5fb6-92af-ade3-8ef6cf526018@10.8.27.9@o2ib6:27/0 lens 480/568 e 0 to 0 dl 1562974017 ref 2 fl Interpret:/0/0 rc 0/0 Jul 12 16:26:52 fir-md1-s1 kernel: Lustre: 10585:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jul 12 16:27:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 26a3bc1d-bdb0-bbb4-3006-c88ecc2f97cd (at 10.9.0.62@o2ib4) Jul 12 16:27:21 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 12 16:27:27 fir-md1-s1 kernel: LustreError: 23749:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562973957, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f3312bbcc80/0x5d9ee63c05ae18f3 lrc: 3/1,0 mode: --/PR res: [0x2c002c34d:0x696:0x0].0x0 bits 0x13/0x0 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23749 timeout: 0 lvb_type: 0 Jul 12 16:27:27 fir-md1-s1 kernel: LustreError: 23749:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 59 previous similar messages Jul 12 16:27:32 fir-md1-s1 kernel: LustreError: 21428:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1562973962, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f182b4a3180/0x5d9ee63c05b1a70f lrc: 3/1,0 mode: --/PR res: [0x2c002c34d:0x696:0x0].0x0 bits 0x13/0x0 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21428 timeout: 0 lvb_type: 0 Jul 12 16:27:32 fir-md1-s1 kernel: LustreError: 21428:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 12 16:27:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 8c306b25-6991-df5d-1f1e-98e88c217f74 (at 10.8.27.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3995310400, cur 1562974056 expire 1562973906 last 1562973829 Jul 12 16:27:36 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 16:27:37 fir-md1-s1 kernel: LustreError: 23704:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f2ecb194b00 x1636731505256496/t0(0) o104->fir-MDT0000@10.8.27.7@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 12 16:28:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 16:28:43 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 12 16:30:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 16:30:29 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 16:36:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 16:36:26 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 12 16:37:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 16:37:49 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 12 16:39:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 16:39:06 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 16:41:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 16:41:50 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 12 16:47:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 16:47:51 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 12 16:49:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 16:49:15 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 16:50:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 16:50:59 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 16:52:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 16:52:02 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 12 16:57:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 16:57:53 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 12 16:59:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 16:59:20 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 17:00:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client f8cb00db-6694-c576-4092-4a678c6e80f9 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f44ec62a400, cur 1562976004 expire 1562975854 last 1562975777 Jul 12 17:00:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 17:01:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 17:01:59 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 17:02:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 17:02:30 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 12 17:08:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 17:08:11 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 12 17:09:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 12 17:09:36 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 12 17:12:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 17:12:46 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 12 17:15:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 17:15:07 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 17:18:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 17:18:29 fir-md1-s1 kernel: Lustre: Skipped 117 previous similar messages Jul 12 17:20:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 17:20:17 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 17:23:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 17:23:31 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 12 17:28:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 17:28:47 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 12 17:30:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 17:30:30 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 12 17:33:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 17:33:18 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 17:33:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 17:33:43 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 17:38:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 17:38:53 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 12 17:41:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 17:41:11 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 12 17:44:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 17:44:55 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 12 17:49:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 17:49:05 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 17:50:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 17:50:48 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 12 17:51:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 17:51:45 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 12 17:55:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 17:55:01 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 12 17:57:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client f1b26272-cb99-9dbe-fdc3-6a70f1d77cbb (at 10.9.112.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f147812a000, cur 1562979472 expire 1562979322 last 1562979245 Jul 12 17:57:52 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 17:59:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 17:59:10 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 18:01:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 18:01:51 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 18:05:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 18:05:23 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 12 18:07:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 18:07:40 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 18:09:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 12 18:09:18 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 12 18:12:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 18:12:47 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 18:15:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 18:15:42 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 12 18:18:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 18:19:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 18:19:19 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 12 18:23:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 18:23:19 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 18:26:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 12 18:26:44 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 12 18:29:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 18:29:24 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 12 18:29:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 18:29:38 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 12 18:35:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 18:35:01 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 12 18:36:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 18:36:47 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 12 18:39:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 18:39:25 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 12 18:41:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client dab487b8-ac88-5102-eda2-bdced899b20d (at 10.8.8.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2bcecf6000, cur 1562982092 expire 1562981942 last 1562981865 Jul 12 18:41:32 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 12 18:45:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 18:45:20 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 18:46:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 18:46:50 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 18:49:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 18:49:28 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 12 18:50:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 18:50:16 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 18:55:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 18:55:23 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 18:56:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 18:56:50 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 12 18:59:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 18:59:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 18:59:29 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jul 12 19:05:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 19:05:27 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 19:07:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 19:07:23 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 12 19:09:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 19:09:37 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 12 19:10:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 19:15:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 19:15:45 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 12 19:17:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 19:17:25 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 19:19:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 19:19:42 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 12 19:22:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 19:22:13 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 19:26:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 19:26:36 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 19:28:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 19:28:01 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 12 19:28:36 fir-md1-s1 kernel: Lustre: 23651:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 Jul 12 19:29:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 19:29:51 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 12 19:38:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 19:38:33 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 19:38:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 19:38:50 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 19:39:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 19:39:51 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 19:48:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 19:48:43 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 19:50:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 19:50:07 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 12 19:50:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 19:50:07 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 12 19:58:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 19:58:46 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 12 20:00:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 20:00:08 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 12 20:00:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 20:00:08 fir-md1-s1 kernel: Lustre: Skipped 132 previous similar messages Jul 12 20:01:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 20:01:02 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 12 20:09:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 20:09:24 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 20:10:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 20:10:19 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 12 20:10:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 20:10:19 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Jul 12 20:18:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 20:18:16 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 20:19:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 20:19:24 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 12 20:20:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 20:20:20 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 20:20:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 20:20:20 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 12 20:20:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 20:27:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 20:29:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 20:29:31 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 12 20:30:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 20:30:25 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 12 20:30:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 20:31:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 20:31:27 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 12 20:39:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 20:39:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 20:39:54 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 12 20:40:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 20:40:35 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 12 20:42:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 20:42:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 20:42:05 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 12 20:48:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 20:50:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 20:50:12 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 20:50:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 20:50:39 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 12 20:53:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 20:53:08 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 12 21:00:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 21:00:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 21:00:43 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 21:00:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 21:00:43 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 12 21:03:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 21:03:13 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 12 21:10:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 21:10:47 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 12 21:10:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 21:10:50 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 12 21:13:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 21:13:27 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 21:16:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 21:21:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 21:21:26 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 12 21:21:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 21:21:27 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 21:23:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 21:23:31 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 12 21:29:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 21:31:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 21:31:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 21:31:37 fir-md1-s1 kernel: Lustre: Skipped 126 previous similar messages Jul 12 21:31:37 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 12 21:33:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 21:33:55 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 12 21:34:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5c4c5b6a-001d-e26d-f4d4-23e598bc49a5 (at 10.9.103.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4518949800, cur 1562992498 expire 1562992348 last 1562992271 Jul 12 21:34:58 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 21:40:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 21:40:22 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 12 21:41:30 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1f71ffc000, cur 1562992890 expire 1562992740 last 1562992663 Jul 12 21:41:30 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 12 21:41:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 21:41:40 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 21:41:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 21:41:57 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 12 21:44:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 21:44:03 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 12 21:51:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 21:51:46 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 12 21:52:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 21:52:10 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 12 21:54:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 21:54:25 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 21:54:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 21:54:38 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 22:02:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 22:02:05 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 12 22:02:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bf2fec24-a441-2b9a-3334-0bc96ce2df5f (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2255dbd400, cur 1562994130 expire 1562993980 last 1562993903 Jul 12 22:02:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 22:02:29 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 22:04:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 22:04:32 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 12 22:12:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 22:12:15 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 12 22:13:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 22:13:17 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 12 22:15:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 22:15:29 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 22:19:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 22:19:50 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 22:22:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 22:22:18 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 12 22:23:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 22:23:25 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 12 22:26:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 22:26:28 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 12 22:32:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 22:32:19 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 12 22:33:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 12 22:33:46 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 22:36:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 22:36:29 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 12 22:42:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 12 22:42:30 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 12 22:43:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 22:43:52 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 12 22:47:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 12 22:47:33 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 12 22:52:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 22:52:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 12 22:52:30 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 12 22:53:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 22:53:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 22:53:57 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 12 22:56:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 22:59:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 22:59:55 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 12 23:02:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 23:02:36 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 12 23:03:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 12 23:03:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 23:03:59 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 12 23:08:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 23:10:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 23:10:40 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 12 23:13:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 23:13:11 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 12 23:14:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 23:15:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 12 23:15:02 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 23:17:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 74e11ba4-980c-f875-a68c-e22360c64935 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2fa668f800, cur 1562998668 expire 1562998518 last 1562998441 Jul 12 23:17:48 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 23:21:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 23:21:17 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 12 23:23:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 23:23:19 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 12 23:25:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 23:25:04 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 12 23:27:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 23:27:22 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 12 23:31:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 12 23:31:51 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 12 23:32:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ef2b19e1-b66e-f78f-ca40-ca13fb6d4d06 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24b6f9e000, cur 1562999520 expire 1562999370 last 1562999293 Jul 12 23:32:00 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 12 23:33:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 23:33:24 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 12 23:35:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 23:35:15 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 12 23:38:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 23:38:07 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 23:42:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 12 23:42:08 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 12 23:43:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 12 23:43:46 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 12 23:45:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 12 23:45:19 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 12 23:51:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 12 23:51:41 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 12 23:52:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 12 23:52:56 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 12 23:53:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 12 23:53:54 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 12 23:55:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 12 23:55:20 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 00:02:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 00:02:49 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 00:02:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 00:02:58 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 00:04:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 00:04:07 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 13 00:05:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 00:05:29 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 00:13:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 00:13:17 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 13 00:14:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 00:14:12 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 00:15:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 00:15:32 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 13 00:20:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 00:20:03 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 00:24:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 00:24:25 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 13 00:25:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 00:25:38 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 00:27:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 00:27:42 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 13 00:31:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 00:31:12 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 00:34:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 00:34:31 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 00:35:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 00:35:42 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 13 00:37:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 00:37:43 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 00:44:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 00:44:33 fir-md1-s1 kernel: Lustre: Skipped 102 previous similar messages Jul 13 00:45:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 00:45:33 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 00:45:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 00:45:53 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 00:47:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 00:47:45 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 13 00:54:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 00:54:35 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 13 00:56:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 00:56:10 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 00:56:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 00:56:38 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 00:57:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 00:57:48 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 13 01:04:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 01:04:35 fir-md1-s1 kernel: Lustre: Skipped 117 previous similar messages Jul 13 01:06:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 01:06:15 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 01:08:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 01:08:18 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 01:08:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client b39a470f-e258-0a6b-08d8-9a798f8b9f1c (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fd37d000, cur 1563005318 expire 1563005168 last 1563005091 Jul 13 01:08:38 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 01:08:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 01:08:40 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 01:14:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 01:14:37 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 01:16:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 01:16:41 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 13 01:19:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 01:19:31 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 01:24:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 01:24:42 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 13 01:26:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5d399386-b1fb-d405-e88f-f20c8d175a51 (at 10.8.25.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f252367b000, cur 1563006385 expire 1563006235 last 1563006158 Jul 13 01:26:25 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 01:26:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 01:26:46 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 13 01:28:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 01:28:22 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 13 01:30:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 13 01:30:11 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 13 01:34:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 01:34:53 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 13 01:36:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 01:36:53 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 13 01:40:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ef4e1fdc-8937-844b-21bf-e4b85d8fcd3a (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f28de8a9c00, cur 1563007208 expire 1563007058 last 1563006981 Jul 13 01:40:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 01:41:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 13 01:41:48 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 01:44:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 01:44:59 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 13 01:45:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 01:45:18 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 13 01:47:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 01:47:10 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 13 01:51:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 01:51:57 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 13 01:55:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 01:55:03 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 13 01:57:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 01:57:18 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 01:59:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 01:59:22 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 13 02:02:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 02:02:54 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 02:05:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 02:05:07 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 02:05:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f0fdbf61400, cur 1563008715 expire 1563008565 last 1563008488 Jul 13 02:05:15 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 02:07:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 02:07:22 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 02:12:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 992f0c36-535d-31fb-df55-36ff304cdd4d (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2ef4fee800, cur 1563009133 expire 1563008983 last 1563008906 Jul 13 02:12:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 02:12:53 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 02:13:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 02:13:23 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 13 02:15:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 02:15:14 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 13 02:17:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 02:17:32 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 13 02:24:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 13 02:24:38 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 02:25:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 02:25:17 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 13 02:26:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 02:27:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 02:27:34 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 02:34:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 02:34:42 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 02:35:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 02:35:26 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 02:37:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 02:37:42 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 02:40:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 02:40:01 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 13 02:45:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 02:45:30 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 13 02:46:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 02:46:25 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 13 02:47:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 02:47:52 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 02:50:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client f9af0770-c7bd-566c-affe-31bdf8c8eed6 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4535c7ec00, cur 1563011445 expire 1563011295 last 1563011218 Jul 13 02:50:45 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 02:56:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 02:56:03 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 13 02:57:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 02:57:22 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 13 02:58:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 02:58:26 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 02:59:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client b5210801-eaf6-299e-958d-1d0d0937fe0b (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f31f0194800, cur 1563011964 expire 1563011814 last 1563011737 Jul 13 02:59:24 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 03:04:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 03:06:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 03:06:14 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 03:08:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 03:08:29 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 13 03:08:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 03:08:53 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 03:16:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 03:16:10 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 13 03:16:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 03:16:15 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 03:19:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 03:19:06 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 03:19:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 03:19:16 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 03:26:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 03:26:15 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 13 03:26:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 03:29:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 13 03:29:43 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 13 03:29:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 03:29:47 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 03:35:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 03:35:23 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 03:36:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 03:36:24 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 03:40:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 03:40:24 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 03:40:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 03:40:44 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 03:46:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 03:46:45 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 13 03:50:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 03:50:41 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 13 03:50:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 03:50:52 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 13 03:56:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 03:56:55 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 13 04:00:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 04:00:33 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 13 04:01:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 04:01:00 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 04:01:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 04:01:33 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 04:03:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 04:04:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client ecb5ad2c-7f68-a999-a141-4ba5fa8d702a (at 10.8.13.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2508cc5000, cur 1563015893 expire 1563015743 last 1563015666 Jul 13 04:04:53 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 04:06:33 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3b8a762400, cur 1563015993 expire 1563015843 last 1563015766 Jul 13 04:06:33 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 13 04:07:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 04:07:27 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 04:11:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 04:11:06 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 04:12:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 04:12:02 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 13 04:14:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 04:17:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 04:17:33 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 13 04:19:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 04:19:50 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 13 04:21:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 04:21:58 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 04:22:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 13 04:22:35 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 13 04:27:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 76e12dce-c40a-5a48-4e41-308e77527a3a (at 10.8.30.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f250d9c7400, cur 1563017223 expire 1563017073 last 1563016996 Jul 13 04:27:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 76e12dce-c40a-5a48-4e41-308e77527a3a (at 10.8.30.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f250da2b800, cur 1563017243 expire 1563017093 last 1563017016 Jul 13 04:27:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 13 04:27:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 04:27:49 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 13 04:30:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 04:30:37 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 04:32:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 04:32:07 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 13 04:32:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 04:32:48 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 04:38:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 04:38:02 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 13 04:40:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 04:40:38 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 13 04:42:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 04:42:10 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 04:42:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 04:42:49 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 13 04:47:41 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f39a3bc2c00, cur 1563018461 expire 1563018311 last 1563018234 Jul 13 04:48:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 04:48:08 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 13 04:51:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 04:51:12 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 13 04:53:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 04:53:05 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 13 04:53:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 04:53:53 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 04:58:39 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 04:58:39 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 13 05:02:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 05:02:03 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 13 05:03:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 05:03:05 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 05:03:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 05:03:53 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 13 05:08:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 05:08:47 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 05:12:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 05:12:27 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 13 05:13:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 05:13:24 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 05:15:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 05:15:30 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 13 05:19:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 05:19:12 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 05:23:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 05:23:02 fir-md1-s1 kernel: LustreError: Skipped 15 previous similar messages Jul 13 05:23:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2b07aab000, cur 1563020593 expire 1563020443 last 1563020366 Jul 13 05:24:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 05:24:23 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 05:25:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 05:25:45 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 13 05:29:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 05:29:16 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 13 05:33:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 05:33:04 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 13 05:34:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 05:34:27 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 05:36:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 05:36:50 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 13 05:39:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 05:39:25 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 13 05:43:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 05:43:14 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 13 05:44:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 05:44:39 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 05:46:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 05:46:55 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 13 05:49:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 05:49:34 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 13 05:53:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 05:53:20 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 13 05:55:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 05:55:10 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 13 05:57:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 05:57:00 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 13 05:59:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 05:59:51 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 13 06:03:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 06:03:41 fir-md1-s1 kernel: LustreError: Skipped 13 previous similar messages Jul 13 06:05:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 06:05:30 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 06:07:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 06:07:01 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 06:10:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 06:10:02 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 13 06:14:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 06:14:43 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 13 06:15:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 06:15:31 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 13 06:17:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 06:17:15 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 13 06:20:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 06:20:04 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 13 06:24:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 06:24:52 fir-md1-s1 kernel: LustreError: Skipped 13 previous similar messages Jul 13 06:25:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 06:25:32 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 06:27:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 13 06:27:37 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 06:30:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 06:30:24 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 06:32:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5efe2f72-0a5c-fff4-b523-16ec38cac7e2 (at 10.9.103.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f252147f400, cur 1563024778 expire 1563024628 last 1563024551 Jul 13 06:35:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 06:35:25 fir-md1-s1 kernel: LustreError: Skipped 13 previous similar messages Jul 13 06:35:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 06:35:45 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 06:38:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 06:38:55 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 06:40:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 06:40:27 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 13 06:45:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 06:45:59 fir-md1-s1 kernel: LustreError: Skipped 14 previous similar messages Jul 13 06:46:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 06:46:00 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 06:49:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 06:49:16 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 13 06:50:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 06:50:48 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 06:56:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 06:56:07 fir-md1-s1 kernel: LustreError: Skipped 13 previous similar messages Jul 13 06:56:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 06:56:20 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 07:00:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 07:00:41 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 13 07:01:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 07:01:12 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 13 07:06:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 07:06:20 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 13 07:07:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 07:07:00 fir-md1-s1 kernel: LustreError: Skipped 16 previous similar messages Jul 13 07:10:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 07:10:47 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 13 07:11:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 07:11:14 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 13 07:16:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 07:16:27 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 13 07:17:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 07:17:43 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 13 07:21:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 07:21:31 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 13 07:21:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 07:21:31 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 13 07:26:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 07:26:59 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 07:27:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 07:27:48 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 13 07:29:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3266f5d400, cur 1563028179 expire 1563028029 last 1563027952 Jul 13 07:29:39 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 07:31:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 07:31:38 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 13 07:31:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 07:31:40 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 13 07:37:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 07:37:12 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 07:39:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 07:39:15 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 13 07:41:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 07:41:44 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 07:41:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 07:41:44 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 13 07:47:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 07:47:25 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 07:50:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 07:50:07 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 13 07:51:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 07:51:49 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 13 07:51:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 07:51:52 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 13 07:57:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 07:57:58 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 13 08:02:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 08:02:04 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 08:02:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 08:02:19 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 08:03:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 282d29b1-b17a-d2c6-8c52-58515f7a3b2a (at 10.9.101.38@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2d7fa6b000, cur 1563030208 expire 1563030058 last 1563029981 Jul 13 08:04:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 08:04:08 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 13 08:08:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 08:08:12 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 13 08:12:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 08:12:17 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 13 08:13:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 08:13:02 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 13 08:14:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 08:14:14 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 13 08:18:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 08:18:14 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 08:22:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 08:22:22 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 13 08:24:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 08:24:09 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 08:24:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 08:24:15 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 13 08:28:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 08:28:25 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 08:32:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 08:32:25 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jul 13 08:34:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 08:34:17 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 13 08:34:34 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 08:34:34 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 08:38:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 08:38:48 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 13 08:43:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 08:43:02 fir-md1-s1 kernel: Lustre: Skipped 110 previous similar messages Jul 13 08:44:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 13 08:44:24 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 13 08:47:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 08:47:27 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 13 08:48:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 08:48:48 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 08:53:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 08:53:10 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 13 08:55:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 08:55:02 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 13 08:58:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 08:58:29 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 08:58:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 08:58:51 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 09:03:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 09:03:14 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 13 09:04:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2b1ced5c00, cur 1563033890 expire 1563033740 last 1563033663 Jul 13 09:04:50 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 13 09:05:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 09:05:03 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 13 09:08:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 09:08:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 09:08:52 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 09:08:52 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 09:13:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 09:13:49 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 13 09:15:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 09:15:06 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 09:19:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 09:19:15 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 09:20:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 09:20:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 09:20:15 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 13 09:23:50 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 09:23:50 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 13 09:26:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 09:26:41 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 09:29:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 09:29:19 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 09:31:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 09:31:26 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 09:34:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 09:34:28 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 09:37:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 13 09:37:20 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 09:39:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 09:39:59 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 09:42:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 09:42:24 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 13 09:44:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 09:44:52 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 13 09:48:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 09:48:16 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 09:50:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 09:50:01 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 09:52:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 09:52:50 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 09:54:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 09:54:56 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 13 09:58:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 09:58:17 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 13 10:01:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 10:01:11 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 10:03:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 10:03:54 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 10:05:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 10:05:15 fir-md1-s1 kernel: Lustre: Skipped 105 previous similar messages Jul 13 10:08:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 10:08:27 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 13 10:11:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 10:11:24 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 10:14:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 10:14:51 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 10:15:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 10:15:21 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 13 10:18:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 10:18:33 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 13 10:21:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 10:21:32 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 13 10:25:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 10:25:20 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 13 10:25:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 10:25:27 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jul 13 10:31:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 10:31:23 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 10:31:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 10:31:43 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 10:35:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 10:35:24 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 10:35:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 10:35:27 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 10:42:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 10:42:08 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 13 10:42:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 10:42:10 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 10:45:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 10:45:30 fir-md1-s1 kernel: Lustre: Skipped 102 previous similar messages Jul 13 10:47:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 10:47:41 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 10:52:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 10:52:13 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 10:52:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 10:52:17 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 13 10:55:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 10:55:31 fir-md1-s1 kernel: Lustre: Skipped 106 previous similar messages Jul 13 10:59:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 10:59:58 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 11:02:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 11:02:53 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 11:04:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 11:04:06 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 11:06:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 11:06:33 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 13 11:11:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 11:11:43 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 11:13:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 11:13:57 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 13 11:14:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 11:14:07 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 11:16:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 11:16:35 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 13 11:22:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 11:22:12 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 13 11:24:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 11:24:18 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 11:24:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 11:24:22 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 11:26:41 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 11:26:41 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 11:33:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 11:33:17 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 11:34:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 11:34:30 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 11:34:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 11:34:53 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 13 11:37:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 11:37:02 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 11:43:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 11:43:20 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 11:44:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 11:44:30 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 11:44:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 11:44:55 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 11:47:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 11:47:03 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 13 11:54:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 11:54:43 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 11:54:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 11:54:45 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 11:55:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 11:55:30 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 13 11:57:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 11:57:16 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Jul 13 12:04:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 12:04:49 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 12:05:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 12:05:13 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 12:06:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 12:06:52 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 12:07:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 12:07:20 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 13 12:15:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 12:15:08 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 12:16:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 12:16:49 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 12:17:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 12:17:22 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 12:17:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 13 12:17:39 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 13 12:25:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 12:25:21 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 12:27:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 12:27:21 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 12:27:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 12:27:23 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 13 12:27:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 12:27:49 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 13 12:35:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 12:35:26 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 12:37:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 12:37:23 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 12:37:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 12:37:26 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 13 12:37:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 12:37:55 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 13 12:46:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 12:46:08 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 12:47:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 12:47:29 fir-md1-s1 kernel: Lustre: Skipped 103 previous similar messages Jul 13 12:47:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 12:47:35 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 13 12:48:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 12:48:00 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 13 12:56:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 12:56:27 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 13 12:57:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 12:57:49 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 13 12:58:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 12:58:07 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 13 12:59:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 12:59:16 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 13:06:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 13:06:50 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 13:07:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 13:07:57 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 13 13:09:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 13:09:34 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 13 13:13:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 13:13:00 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 13:17:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 13:17:05 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 13 13:18:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 13:18:21 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 13 13:20:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 13:20:36 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 13:23:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 13:23:33 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 13:27:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 13:27:23 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 13:28:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 13:28:25 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 13:32:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 13:32:08 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 13 13:33:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 13:33:49 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 13:37:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 13:37:34 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 13:38:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 13:38:27 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 13 13:42:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 13:42:14 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 13:43:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 13:43:53 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 13 13:47:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 13:47:56 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 13 13:48:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 13:48:36 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 13 13:52:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 13:52:29 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 13 13:55:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 13:55:47 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 13:58:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 13:58:08 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 13 13:59:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 13:59:01 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 14:00:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1d90ec5000, cur 1563051623 expire 1563051473 last 1563051396 Jul 13 14:03:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 14:03:12 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 13 14:06:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 14:06:11 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 14:08:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 14:08:19 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 14:09:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 14:09:12 fir-md1-s1 kernel: Lustre: Skipped 97 previous similar messages Jul 13 14:13:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 14:13:16 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 14:16:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 14:16:32 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 13 14:18:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 14:18:20 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 14:19:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 14:19:19 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 13 14:24:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 14:24:53 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 14:27:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 14:27:33 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 14:28:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 14:28:54 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 13 14:29:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 14:29:24 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 13 14:30:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2e9ca01000, cur 1563053435 expire 1563053285 last 1563053208 Jul 13 14:35:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 14:35:42 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 14:37:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 14:37:38 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 14:39:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 14:39:07 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 14:39:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 14:39:29 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 13 14:45:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 14:45:43 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 14:47:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 14:47:50 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 14:49:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 14:49:23 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 14:49:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 14:49:55 fir-md1-s1 kernel: Lustre: Skipped 118 previous similar messages Jul 13 14:55:59 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 14:55:59 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 14:58:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 14:58:21 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 13 14:59:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 14:59:43 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 13 15:00:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 15:00:06 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 13 15:06:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 15:06:55 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 15:08:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 15:08:42 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 15:09:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 15:09:59 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 15:10:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 15:10:06 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 13 15:19:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 15:19:36 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 13 15:20:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 15:20:07 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 13 15:20:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 15:20:18 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 15:21:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 15:21:40 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 15:29:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 15:29:46 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 13 15:30:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 15:30:17 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 13 15:30:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 15:30:37 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 15:32:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 15:32:38 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 13 15:39:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 15:39:51 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 13 15:40:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 15:40:17 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 13 15:40:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 15:40:46 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 13 15:42:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 15:42:45 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 15:49:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 15:49:56 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 15:50:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 15:50:18 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 13 15:50:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 15:50:55 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 13 15:53:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 15:53:35 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 16:00:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 16:00:02 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 13 16:00:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 16:00:28 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 13 16:01:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 16:01:05 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 16:03:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 16:03:35 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 16:10:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 16:10:04 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 13 16:11:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 16:11:15 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 16:11:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 16:11:25 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 16:17:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 16:17:00 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 16:20:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 16:20:04 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 13 16:21:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 16:21:16 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 13 16:21:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 16:21:29 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 13 16:31:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 16:31:18 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 13 16:31:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 16:31:43 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 16:32:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 16:32:11 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 13 16:37:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 16:37:06 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 13 16:41:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 16:41:19 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 13 16:41:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 16:41:49 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 13 16:42:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 16:42:47 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 13 16:51:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 16:51:16 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 16:51:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 16:51:51 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 16:51:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 16:51:51 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 13 16:52:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 16:52:47 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 17:01:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 17:01:25 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 17:01:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 17:01:55 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 13 17:02:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 17:02:12 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 13 17:03:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 17:03:08 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 13 17:11:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 17:11:55 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 13 17:11:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 17:11:56 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 13 17:12:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 17:12:22 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 17:13:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 17:13:08 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 17:22:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 17:22:15 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 13 17:22:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 17:22:28 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 13 17:22:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 17:22:32 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 13 17:22:37 fir-md1-s1 kernel: Lustre: 23633:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563063750/real 1563063750] req@ffff8f345af3b900 x1636732585657632/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563063757 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 13 17:22:37 fir-md1-s1 kernel: Lustre: 23633:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 13 17:25:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 17:25:25 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 13 17:28:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2522e43400, cur 1563064091 expire 1563063941 last 1563063864 Jul 13 17:32:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 17:32:18 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 13 17:32:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 17:32:47 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 13 17:33:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 17:33:39 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 17:35:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 17:35:39 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 17:42:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 17:42:20 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 13 17:43:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 17:43:51 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 17:44:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 17:44:45 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 13 17:44:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 60f59e7b-5296-e995-71c3-01213d30e8c4 (at 10.8.24.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34f1ed3800, cur 1563065090 expire 1563064940 last 1563064863 Jul 13 17:46:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 17:46:25 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 13 17:52:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 17:52:36 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 13 17:53:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 17:53:55 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 17:55:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 17:55:55 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 13 17:58:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 17:58:42 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 18:03:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 18:03:06 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 13 18:03:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 18:03:57 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 18:05:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 18:05:58 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 13 18:09:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 18:09:53 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 18:13:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 18:13:06 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 13 18:13:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 18:13:59 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 18:21:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 18:21:21 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 13 18:23:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 18:23:12 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 13 18:24:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 18:24:06 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 13 18:31:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 18:31:22 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 13 18:33:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 18:33:20 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 13 18:34:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 18:34:40 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 18:35:20 fir-md1-s1 kernel: Lustre: 23642:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f1f2c4a0c00 x1634178254817744/t352284552582(0) o36->185d31e3-2aa7-c8dc-f4ab-116af2588723@10.9.109.14@o2ib4:25/0 lens 488/3152 e 1 to 0 dl 1563068125 ref 2 fl Interpret:/0/0 rc 0/0 Jul 13 18:40:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 18:40:07 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 13 18:41:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 18:41:26 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 13 18:43:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 18:43:20 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 13 18:44:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 18:44:45 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 18:52:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 18:52:11 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 13 18:53:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 18:53:33 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 13 18:54:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 18:54:50 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 18:55:01 fir-md1-s1 kernel: Lustre: 21411:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563069294/real 1563069294] req@ffff8f0ec188c500 x1636732624695296/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563069301 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 13 18:58:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 19:02:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 19:02:14 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 19:03:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 19:03:54 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 13 19:04:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 19:04:51 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 13 19:09:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 19:10:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 19:13:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 19:13:27 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 13 19:13:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 19:13:56 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 13 19:15:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 19:15:00 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 19:18:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 19:23:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 19:23:31 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 13 19:23:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 19:24:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 19:24:23 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 13 19:25:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 19:25:17 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 19:31:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 19:31:51 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 13 19:33:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 19:33:35 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 19:34:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 19:34:42 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 13 19:35:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 19:35:23 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 13 19:39:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 19:43:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 19:43:38 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 13 19:44:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 19:44:42 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 13 19:45:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 19:45:26 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 19:54:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 19:54:01 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 19:54:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 19:54:49 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 13 19:55:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 19:55:26 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 20:00:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 20:04:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 20:04:31 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 20:04:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 20:04:55 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 13 20:05:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 20:05:37 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 13 20:13:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 20:13:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 20:15:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 20:15:15 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 13 20:15:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 20:15:52 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Jul 13 20:16:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 20:16:10 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 13 20:25:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 20:25:23 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 20:26:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 20:26:16 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 20:28:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 13 20:28:16 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Jul 13 20:32:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 20:35:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 20:35:28 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 20:36:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 20:36:47 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 13 20:39:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 20:39:31 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 20:45:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 20:45:42 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 13 20:47:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 20:47:21 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 13 20:50:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 13 20:50:55 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 13 20:55:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 20:55:44 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 13 20:57:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 20:57:35 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 21:01:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 21:01:04 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 13 21:05:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 21:05:45 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 13 21:07:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 21:07:39 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 13 21:11:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 21:11:08 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 21:16:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 21:16:04 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 13 21:17:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 21:17:47 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 21:21:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 21:21:25 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 13 21:26:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 21:26:15 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 21:26:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 21:27:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 21:27:57 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 21:31:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 21:31:27 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 21:36:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 21:36:16 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 13 21:38:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 21:38:02 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 13 21:42:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 21:42:31 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 13 21:46:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 21:46:29 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 21:46:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 21:47:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 21:48:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 21:48:27 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 21:52:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 21:52:34 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 21:56:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 21:56:31 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 13 21:58:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 21:58:42 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 13 22:02:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 22:02:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 22:02:45 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 13 22:07:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 22:07:11 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 13 22:08:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 22:08:48 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 13 22:11:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 22:13:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 22:13:48 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 13 22:17:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 13 22:17:22 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 13 22:18:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 22:18:55 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 13 22:24:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 22:24:19 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 13 22:27:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 22:27:26 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 13 22:28:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 22:28:57 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 13 22:29:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 22:35:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 13 22:35:09 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 22:37:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 22:37:34 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 13 22:39:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 13 22:39:02 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 13 22:46:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 22:46:17 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 13 22:47:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 13 22:47:35 fir-md1-s1 kernel: Lustre: Skipped 106 previous similar messages Jul 13 22:48:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 22:49:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 13 22:49:21 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 22:56:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 22:56:30 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 13 22:57:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 22:57:42 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 13 22:58:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 22:59:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 22:59:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 22:59:47 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 13 23:03:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 23:06:59 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 23:06:59 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 13 23:07:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 13 23:07:56 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 23:10:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 13 23:10:08 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 13 23:17:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 13 23:17:06 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 23:18:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 23:18:20 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 13 23:20:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 23:20:11 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 13 23:27:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 23:27:11 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 13 23:28:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 23:28:21 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 13 23:30:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 13 23:30:13 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 13 23:37:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 23:37:14 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 13 23:38:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 13 23:38:48 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 13 23:40:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 23:40:25 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 13 23:47:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 13 23:47:50 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 13 23:49:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 23:49:01 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 13 23:49:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 23:50:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 13 23:50:36 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 13 23:53:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client dc206ad9-6c70-6097-3407-cb9490b12136 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fc2bf000, cur 1563087236 expire 1563087086 last 1563087009 Jul 13 23:53:56 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 13 23:55:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 13 23:57:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 13 23:57:54 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 13 23:59:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 13 23:59:08 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Jul 14 00:00:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 00:00:38 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 00:07:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 00:07:54 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 00:09:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 00:09:51 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 14 00:10:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 00:10:50 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 00:18:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 00:18:14 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 00:19:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 00:19:56 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 14 00:21:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 00:21:08 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 00:25:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 00:28:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 00:28:22 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 14 00:29:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 00:29:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 00:29:59 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 14 00:31:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 00:31:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 00:31:19 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 00:36:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 00:38:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 00:38:26 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 14 00:40:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 00:40:24 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 14 00:42:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 00:42:24 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 00:48:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 00:48:47 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 14 00:50:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 00:50:31 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 14 00:52:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 00:52:29 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 00:55:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 00:56:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 00:59:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 00:59:17 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 14 01:00:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 01:00:45 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 14 01:02:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 01:02:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 01:02:57 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 01:03:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 01:10:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 01:10:58 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 14 01:12:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 01:12:39 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 01:12:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 01:12:59 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 01:20:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 01:20:59 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 14 01:23:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 01:23:04 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 14 01:27:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 01:27:19 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 01:30:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 01:31:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 01:31:26 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 14 01:33:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 01:33:15 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 01:37:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 01:37:34 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 14 01:41:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 01:41:30 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 14 01:43:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 01:43:35 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 01:50:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 01:50:53 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 14 01:51:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 01:51:37 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 14 01:53:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 01:53:38 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 01:57:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 01:58:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 01:59:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 02:01:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 02:01:47 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 14 02:01:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 02:01:47 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 14 02:03:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 02:03:49 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 02:11:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 02:11:49 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 02:11:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 02:11:49 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 14 02:12:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 02:13:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 02:13:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 02:13:53 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 02:21:01 fir-md1-s1 kernel: LNetError: 20198:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Jul 14 02:22:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 02:22:14 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 14 02:23:58 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 02:24:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 02:24:00 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 02:24:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 02:24:14 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 14 02:32:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 02:32:17 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 14 02:34:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 02:34:06 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 02:34:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 02:34:22 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 02:36:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2e96f0f000, cur 1563096963 expire 1563096813 last 1563096736 Jul 14 02:36:03 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 14 02:42:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 02:42:32 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 14 02:44:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 02:44:26 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 14 02:44:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 02:44:27 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 14 02:52:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 02:52:41 fir-md1-s1 kernel: Lustre: Skipped 106 previous similar messages Jul 14 02:54:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 02:54:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 02:54:32 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 02:54:32 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 14 03:02:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 03:02:57 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 14 03:05:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 03:05:00 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 14 03:08:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 03:08:40 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 03:12:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 03:12:58 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 14 03:14:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 03:15:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 03:15:16 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 03:16:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 03:18:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 03:18:41 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 14 03:21:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 03:23:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 03:23:23 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 14 03:25:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 03:25:23 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 03:26:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 03:26:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 03:29:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2f787d8800, cur 1563100157 expire 1563100007 last 1563099930 Jul 14 03:29:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 03:29:17 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 14 03:33:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 03:33:24 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 14 03:36:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 03:36:00 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 14 03:37:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 03:39:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 03:40:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 03:42:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 03:42:18 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 14 03:43:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 03:43:48 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 14 03:47:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 03:47:31 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 14 03:53:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 03:53:56 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 14 03:54:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 03:54:23 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 14 03:57:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 03:57:37 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 03:59:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 04:04:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 04:04:03 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 14 04:05:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 04:05:47 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 14 04:07:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 04:07:38 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 04:08:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 04:09:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 04:14:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 04:14:08 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 14 04:14:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 04:15:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 04:15:53 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 14 04:17:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 04:17:51 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 14 04:24:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 04:24:09 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 14 04:26:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 04:26:53 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 14 04:27:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 04:27:54 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 04:34:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 04:34:11 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 14 04:37:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 04:37:03 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 14 04:38:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 04:38:08 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 04:44:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 04:44:19 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 14 04:47:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 04:47:40 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 04:48:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 04:48:39 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 04:51:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 04:54:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 04:54:25 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 14 04:56:30 fir-md1-s1 kernel: Lustre: 21370:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563105383/real 1563105383] req@ffff8f14b0f47b00 x1636732844994640/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563105390 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 04:57:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 04:57:53 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 14 04:58:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 04:58:57 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 05:04:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 05:04:31 fir-md1-s1 kernel: Lustre: Skipped 91 previous similar messages Jul 14 05:07:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 05:07:57 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 14 05:09:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 05:09:05 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 05:14:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 05:15:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 05:15:04 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 14 05:18:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 05:18:05 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 14 05:19:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 05:19:09 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 05:25:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 05:25:09 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 14 05:28:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 05:28:28 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 05:29:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 05:29:21 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 05:35:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 05:35:12 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 14 05:38:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 14 05:38:36 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 14 05:39:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 05:39:25 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 05:41:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 05:42:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 05:45:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 05:45:15 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 14 05:48:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 05:48:53 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 14 05:49:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 05:49:26 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 05:53:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 05:54:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 4815f99e-94fc-2359-c40b-ef5555f91d5e (at 10.9.113.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1028230400, cur 1563108853 expire 1563108703 last 1563108626 Jul 14 05:55:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 05:55:18 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Jul 14 05:56:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 05:59:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 05:59:42 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 05:59:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 05:59:55 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 06:03:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 06:05:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 06:05:26 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 06:06:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 06:09:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 14 06:09:44 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 14 06:09:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 06:09:56 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 06:15:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 06:15:43 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 14 06:20:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 06:20:07 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 06:22:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 06:22:06 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 14 06:22:10 fir-md1-s1 kernel: Lustre: 27319:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563110523/real 1563110523] req@ffff8f0b86c40300 x1636732872785552/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563110530 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 06:25:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 06:25:43 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 14 06:30:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 06:30:33 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 06:32:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 06:32:07 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 14 06:32:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 06:35:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 06:35:46 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 14 06:40:37 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 06:40:37 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 06:42:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 06:42:15 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 14 06:45:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 06:45:53 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 14 06:47:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 06:50:03 fir-md1-s1 kernel: Lustre: 23691:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563112196/real 1563112196] req@ffff8f062633b600 x1636732881916080/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563112203 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 06:50:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 06:50:58 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 06:52:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 06:52:41 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 14 06:56:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 06:56:07 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 14 07:01:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 07:01:03 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 07:02:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 07:02:50 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 14 07:06:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 07:06:14 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 14 07:11:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 07:11:08 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 07:11:34 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 07:14:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 14 07:14:11 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 14 07:16:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 07:16:44 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 07:20:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 07:21:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 07:21:23 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 14 07:24:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 07:24:23 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 07:27:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 07:27:06 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 14 07:30:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 07:31:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 07:31:43 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 07:35:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 14 07:35:04 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 14 07:37:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 07:37:08 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 14 07:37:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 07:41:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 07:41:43 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 14 07:45:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 07:45:32 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 14 07:47:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 07:47:16 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 14 07:48:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 07:48:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 07:51:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 07:51:58 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 07:55:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 07:55:37 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 14 07:57:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 07:57:27 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 14 08:02:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 08:02:35 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 08:06:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 08:06:10 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 08:07:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 08:07:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 08:07:42 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 14 08:08:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 08:13:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 08:13:03 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 08:17:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 08:17:35 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 08:17:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 08:17:56 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 14 08:20:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 08:21:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 08:22:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 08:23:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 08:23:18 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 08:23:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 08:27:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 08:27:48 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 14 08:28:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 08:28:05 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 14 08:29:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 08:33:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 08:33:32 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 08:37:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f207fb8ec00, cur 1563118636 expire 1563118486 last 1563118409 Jul 14 08:37:16 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 14 08:38:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 08:38:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 08:38:07 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 14 08:38:07 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 08:43:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 08:43:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 08:43:33 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 08:48:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 08:48:33 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 14 08:49:28 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 08:49:28 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 08:52:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 08:53:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 08:53:45 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 08:58:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 08:58:35 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 14 08:59:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 14 08:59:54 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 09:03:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 09:03:50 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 09:08:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 09:08:44 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 14 09:09:31 fir-md1-s1 kernel: Lustre: 27321:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563120564/real 1563120564] req@ffff8f14426b6f00 x1636732935944528/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563120571 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 09:10:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 09:10:07 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 14 09:13:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 09:13:58 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 09:14:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 09:15:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 09:16:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 09:18:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 09:18:57 fir-md1-s1 kernel: Lustre: Skipped 112 previous similar messages Jul 14 09:20:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 09:20:08 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 14 09:24:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 09:24:37 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 14 09:29:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 09:29:18 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 14 09:30:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 09:30:21 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Jul 14 09:35:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 09:35:02 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 09:39:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 09:39:18 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 14 09:40:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 09:40:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 09:40:27 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 09:45:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 09:45:09 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 09:48:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 09:49:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 09:49:20 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 14 09:49:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 09:50:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 09:51:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 09:51:08 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 09:55:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 09:55:24 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 09:59:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 09:59:23 fir-md1-s1 kernel: Lustre: Skipped 114 previous similar messages Jul 14 10:02:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 10:02:30 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 14 10:05:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 10:05:26 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 10:09:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 10:09:53 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 10:09:53 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 14 10:10:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 10:12:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 10:12:37 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 10:15:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 10:15:48 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 10:19:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 10:19:56 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 14 10:20:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 10:23:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 10:23:13 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 10:26:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 10:26:24 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 10:30:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 10:30:05 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 14 10:33:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 10:33:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 10:34:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 14 10:34:39 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 10:36:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 10:36:31 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 10:40:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 10:40:09 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 14 10:44:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 10:44:42 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 14 10:46:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 10:46:43 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 10:50:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 10:50:16 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 14 10:54:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 10:54:45 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 14 10:56:03 fir-md1-s1 kernel: Lustre: 23589:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563126955/real 1563126955] req@ffff8f1461a47200 x1636732976480176/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563126962 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 10:57:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 10:57:07 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 10:58:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 235865aa-6c17-ab70-0ed1-9e86f8359a3f (at 10.9.107.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2524653400, cur 1563127128 expire 1563126978 last 1563126901 Jul 14 11:00:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 39fe18a4-a89c-1a84-3eb2-1fc3124ee4a0 (at 10.9.108.28@o2ib4) in 211 seconds. I think it's dead, and I am evicting it. exp ffff8f1476702400, cur 1563127204 expire 1563127054 last 1563126993 Jul 14 11:00:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 14 11:00:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 11:00:18 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 14 11:00:20 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 901eeaec-75a4-1e60-2c55-e9a045a13705 (at 10.9.108.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24e6d08c00, cur 1563127220 expire 1563127070 last 1563126993 Jul 14 11:00:20 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 14 11:04:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 11:04:47 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 14 11:07:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 11:07:24 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 11:10:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 11:10:30 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 14 11:17:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 11:17:20 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 14 11:17:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 11:17:54 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 14 11:19:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:20:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:20:32 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 11:20:32 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 14 11:23:10 fir-md1-s1 kernel: Lustre: 21411:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563128583/real 1563128583] req@ffff8f0a34a8a100 x1636732986989184/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563128590 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 11:23:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:27:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 11:27:23 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 14 11:28:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 11:28:00 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 11:30:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.107.20@o2ib4) Jul 14 11:30:43 fir-md1-s1 kernel: Lustre: Skipped 93 previous similar messages Jul 14 11:33:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:34:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:37:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 11:37:25 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 14 11:38:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 11:38:14 fir-md1-s1 kernel: Lustre: Skipped 49143 previous similar messages Jul 14 11:39:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:40:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:40:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 11:40:44 fir-md1-s1 kernel: Lustre: Skipped 49196 previous similar messages Jul 14 11:45:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:48:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 11:48:10 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 11:48:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 11:48:22 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 11:48:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:50:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:50:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 11:50:44 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 14 11:57:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:58:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 11:58:13 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 14 11:58:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 11:58:25 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 14 11:58:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 11:58:45 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 14 12:00:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 12:00:53 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 12:08:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 14 12:08:34 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 12:08:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 12:08:51 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 12:10:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 12:10:54 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 14 12:12:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 12:12:33 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 14 12:13:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f28a9c2d400, cur 1563131583 expire 1563131433 last 1563131356 Jul 14 12:13:03 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 14 12:18:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 12:18:46 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 12:19:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 12:19:08 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 12:21:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 12:21:28 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 14 12:29:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 12:29:25 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 12:31:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 12:31:13 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 12:31:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 12:31:41 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 14 12:34:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 12:36:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 12:38:03 fir-md1-s1 kernel: Lustre: 23651:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563133076/real 1563133076] req@ffff8f08816fb300 x1636733016828064/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563133083 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 12:39:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 12:39:30 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 14 12:42:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 12:42:11 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 14 12:44:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 12:44:14 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 14 12:49:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 12:49:31 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 12:52:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 12:52:19 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 14 12:54:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 12:54:25 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 14 12:59:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 12:59:41 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 13:02:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 13:02:23 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 13:04:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 13:04:54 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 13:06:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 13:09:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 13:09:55 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 13:12:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 13:12:57 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 14 13:15:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 13:15:03 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 13:20:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 13:20:39 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 13:23:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 13:23:13 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 13:25:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 13:25:03 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Jul 14 13:30:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 13:30:48 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 13:33:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 13:33:26 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 13:33:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 13:33:26 fir-md1-s1 kernel: Lustre: Skipped 130 previous similar messages Jul 14 13:34:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 13:35:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 13:35:35 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 14 13:40:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 13:40:54 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 14 13:41:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 13:42:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 13:44:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 13:44:11 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 14 13:46:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 13:46:42 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 13:51:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 13:51:12 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 14 13:54:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 13:54:19 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 14 13:56:43 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 13:56:43 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 14 14:01:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 14:01:14 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 14:04:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 14:04:46 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 14 14:06:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:07:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 14:07:40 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 14 14:11:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 14:11:43 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 14:14:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 14:14:47 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 14 14:17:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:17:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 14:17:58 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 14 14:21:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 14:21:45 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 14:23:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:24:24 fir-md1-s1 kernel: Lustre: 10502:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563139457/real 1563139457] req@ffff8f0810f4ad00 x1636733071020672/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563139464 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 14:24:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 14:24:48 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 14 14:30:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 14:30:50 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 14 14:32:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 14:32:07 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 14:34:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:35:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to d885ba76-20e7-1b98-2018-7e24c1d853b4 (at 10.8.0.68@o2ib6) Jul 14 14:35:01 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 14 14:35:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.68@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:35:12 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 14 14:35:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:41:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:41:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 14 14:41:30 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 14 14:42:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 14:42:28 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 14:43:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:43:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 14:45:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 14:45:04 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Jul 14 14:51:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 14:51:37 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 14:52:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 14:52:38 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 14:55:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 14:55:33 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 14 15:02:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 15:02:03 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 15:02:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 15:02:38 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 15:05:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 15:05:36 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 14 15:06:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 15:12:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 15:12:41 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 14 15:13:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 14 15:13:29 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 15:16:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 15:16:27 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 14 15:20:32 fir-md1-s1 kernel: Lustre: 23672:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563142825/real 1563142825] req@ffff8f0a68e7e600 x1636733088896816/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563142832 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 15:22:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 15:22:57 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 14 15:24:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 15:24:10 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 14 15:26:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 15:26:30 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 14 15:30:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 15:33:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 15:33:24 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 15:36:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 15:36:57 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 14 15:37:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 14 15:37:22 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 14 15:43:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 15:43:33 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 15:47:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 15:47:02 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 15:48:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 15:48:13 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 15:50:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 15:52:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 15:53:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 15:53:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 15:54:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 15:54:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 15:54:07 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 15:57:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 15:57:07 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 14 15:58:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 15:58:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 15:58:40 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 14 15:59:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 16:00:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 16:01:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 16:03:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 16:03:55 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 14 16:04:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 16:04:14 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 16:07:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 16:07:11 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 14 16:09:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 16:09:01 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 14 16:09:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 16:09:02 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Jul 14 16:14:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 16:14:41 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 16:17:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 16:17:12 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 14 16:19:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 16:19:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 16:19:35 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 14 16:19:35 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 14 16:25:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 16:25:05 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 16:27:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 16:27:12 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 14 16:29:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 16:29:46 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 14 16:35:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 16:35:12 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 16:37:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 16:37:13 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 14 16:39:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 16:39:55 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 14 16:45:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 16:45:23 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 16:47:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 16:47:16 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 14 16:50:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 16:50:24 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 14 16:51:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 16:51:20 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 14 16:52:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 16:52:08 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 14 16:56:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 16:56:06 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 16:57:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 16:57:18 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 14 17:01:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 17:01:30 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 14 17:06:19 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 17:06:19 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 17:07:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 17:07:26 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 14 17:09:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 17:11:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 17:11:36 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 14 17:16:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 17:16:24 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 14 17:17:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 17:17:40 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 14 17:21:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 17:21:38 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 17:25:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 17:26:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 17:26:42 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 14 17:28:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 17:28:00 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 14 17:28:36 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 17:28:36 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 14 17:29:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 17:29:28 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 14 17:31:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 17:33:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 17:33:54 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 14 17:34:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 17:36:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 17:36:53 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 17:38:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 17:38:03 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 14 17:43:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 17:43:55 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 17:47:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 17:47:04 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 17:47:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 17:48:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 17:48:06 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 14 17:54:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 17:54:39 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 17:58:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 17:58:10 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 14 17:58:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 17:58:10 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 14 18:04:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 18:04:45 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 14 18:05:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:08:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 18:08:43 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Jul 14 18:08:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 18:08:43 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 14 18:12:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:13:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:16:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 18:16:02 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 14 18:17:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:18:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 18:18:45 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 14 18:18:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:18:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 18:18:55 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 14 18:26:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 18:26:08 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 14 18:28:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 18:28:46 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 14 18:29:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 18:29:07 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Jul 14 18:33:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:33:33 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1ff2293800, cur 1563154413 expire 1563154263 last 1563154186 Jul 14 18:35:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:36:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 18:36:46 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 18:37:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 1f744ac0-b202-c1be-34d8-15a9e9bcd8e8 (at 10.8.25.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f250593ec00, cur 1563154664 expire 1563154514 last 1563154437 Jul 14 18:39:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 18:39:00 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 14 18:39:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:40:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 18:40:27 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 14 18:45:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:46:01 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:46:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 18:46:49 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 14 18:47:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 18:49:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 18:49:05 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 14 18:50:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 18:50:53 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 18:57:44 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 18:57:44 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 18:59:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 18:59:06 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 14 19:01:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 19:01:16 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 14 19:03:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 19:07:45 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 19:07:45 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 14 19:09:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 19:09:16 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 14 19:12:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 19:12:24 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 14 19:17:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 19:17:50 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 14 19:19:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 19:19:17 fir-md1-s1 kernel: Lustre: Skipped 84 previous similar messages Jul 14 19:23:04 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 19:23:04 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 14 19:28:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 19:28:03 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 19:28:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 19:29:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 19:29:35 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 14 19:33:40 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 19:33:40 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 14 19:38:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 19:38:03 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 14 19:39:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 19:39:37 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Jul 14 19:43:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 19:43:56 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 14 19:48:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 19:48:14 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 14 19:49:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 19:49:42 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 14 19:52:34 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 19:54:10 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 19:54:10 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 14 19:55:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 19:57:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 19:58:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 19:59:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 19:59:14 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 14 19:59:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 19:59:44 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 14 20:01:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:02:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:04:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 20:04:25 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 20:08:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:09:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 20:09:45 fir-md1-s1 kernel: Lustre: Skipped 103 previous similar messages Jul 14 20:12:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 20:12:06 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 14 20:14:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 20:14:36 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 20:14:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:18:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:19:48 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 20:19:48 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 14 20:24:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 14 20:24:10 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 14 20:24:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:24:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 20:24:47 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Jul 14 20:28:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client e3a95d5e-2945-1bb1-dd2c-d936b00a965b (at 10.8.10.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f3c25000, cur 1563161301 expire 1563161151 last 1563161074 Jul 14 20:28:21 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 14 20:30:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 20:30:31 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 14 20:31:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client c55aa85e-9bb5-05f7-715c-4f84fb1a4539 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2fca09b800, cur 1563161493 expire 1563161343 last 1563161266 Jul 14 20:31:33 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 14 20:31:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.22.20@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:31:51 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 14 20:35:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 20:35:00 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 20:35:14 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 20:35:14 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 14 20:40:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 20:40:40 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 14 20:41:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:41:04 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 14 20:45:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 20:45:11 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 14 20:45:20 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 20:45:20 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 20:50:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 20:50:51 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 14 20:53:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 20:53:25 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 14 20:55:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 20:55:21 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 21:00:34 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 21:00:34 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 14 21:01:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 21:01:06 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 14 21:05:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 21:05:29 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 14 21:07:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 21:07:16 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 14 21:11:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 21:11:12 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 14 21:12:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 21:12:38 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 14 21:15:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 21:15:47 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 21:19:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f13c52d4800, cur 1563164394 expire 1563164244 last 1563164167 Jul 14 21:19:54 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 14 21:21:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 21:21:14 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 14 21:24:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 21:24:20 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 14 21:25:22 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 21:25:22 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 14 21:25:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 21:25:59 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 21:31:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 21:31:17 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 14 21:35:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 21:35:23 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 21:35:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2fbcb59c00, cur 1563165330 expire 1563165180 last 1563165103 Jul 14 21:36:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 21:36:12 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 14 21:41:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 14 21:41:29 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 14 21:41:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 21:41:39 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 14 21:45:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 21:45:27 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 21:46:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 21:46:23 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 21:51:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 21:51:41 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 14 21:52:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 21:52:04 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 14 21:55:35 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 21:55:35 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 21:56:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 21:56:34 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 14 22:01:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 22:01:43 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 14 22:03:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 22:03:14 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 14 22:05:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 14 22:05:49 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 22:06:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 22:06:44 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 14 22:11:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 22:11:49 fir-md1-s1 kernel: Lustre: Skipped 105 previous similar messages Jul 14 22:16:12 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 22:16:12 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 14 22:16:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 22:16:55 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 14 22:18:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 22:19:22 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 0ea72b5e-3a2b-5bb2-d7d0-9add5a9dde42 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2ec72a5800, cur 1563167962 expire 1563167812 last 1563167735 Jul 14 22:19:26 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client f66bab8e-e08e-c0b5-8b49-2c1c5ad402c5 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1fa82e2800, cur 1563167966 expire 1563167816 last 1563167739 Jul 14 22:19:26 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 14 22:21:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 22:21:55 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 14 22:26:39 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 22:26:39 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 14 22:26:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 22:26:58 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 22:31:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 22:31:59 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 14 22:37:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 14 22:37:01 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 14 22:39:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 22:39:18 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 14 22:40:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 22:40:05 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 14 22:42:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 22:42:00 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 14 22:44:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 22:47:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 22:47:35 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 14 22:49:37 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 22:49:37 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 14 22:51:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 22:52:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 22:52:02 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 14 22:58:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 22:58:02 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 14 22:59:38 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 22:59:38 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 14 23:02:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 23:02:13 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 14 23:06:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 23:06:48 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 14 23:08:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 14 23:08:23 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 23:10:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 14 23:10:17 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 14 23:12:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 23:12:16 fir-md1-s1 kernel: Lustre: Skipped 83 previous similar messages Jul 14 23:18:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 23:18:31 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 14 23:19:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 23:19:38 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 14 23:20:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 23:20:32 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 14 23:22:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 14 23:22:18 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 14 23:28:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 23:28:44 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 14 23:32:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 23:32:15 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 14 23:32:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 14 23:32:27 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 14 23:35:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 23:35:11 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 14 23:38:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 14 23:38:44 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 14 23:42:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 23:42:21 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 14 23:42:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 14 23:42:53 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 14 23:48:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 23:48:39 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 14 23:48:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 14 23:48:51 fir-md1-s1 kernel: Lustre: Skipped 3715 previous similar messages Jul 14 23:52:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 14 23:52:29 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 14 23:53:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 14 23:53:11 fir-md1-s1 kernel: Lustre: Skipped 3741 previous similar messages Jul 14 23:53:57 fir-md1-s1 kernel: Lustre: 23706:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563173630/real 1563173630] req@ffff8f1d9c98c200 x1636733261436736/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563173637 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 14 23:59:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 14 23:59:05 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 14 23:59:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 14 23:59:24 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 15 00:03:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 00:03:26 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Jul 15 00:03:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 00:03:29 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 15 00:09:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 00:09:44 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 15 00:10:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 00:10:21 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 15 00:13:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 00:13:59 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 15 00:14:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 00:14:21 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 15 00:20:14 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 00:20:14 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 15 00:21:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 00:21:09 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 15 00:24:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 00:24:07 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 15 00:25:00 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 00:25:00 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 15 00:30:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 00:30:21 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 15 00:31:21 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 00:31:21 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 15 00:34:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 00:34:18 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 15 00:35:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 00:35:16 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 15 00:37:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 8a2e0e99-b7e2-2b0e-9dbb-18a669bd784a (at 10.9.105.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f350c6ed000, cur 1563176222 expire 1563176072 last 1563175995 Jul 15 00:40:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 00:40:25 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 15 00:42:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 00:42:25 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 15 00:44:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 00:44:19 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 15 00:46:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 15 00:46:10 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 15 00:50:12 fir-md1-s1 kernel: Lustre: 23561:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563177005/real 1563177005] req@ffff8f0b15b83c00 x1636733279194336/t0(0) o106->fir-MDT0000@10.8.30.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563177012 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 15 00:50:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 00:50:26 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 00:52:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 00:52:50 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 15 00:54:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 00:54:50 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 15 00:56:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 00:56:21 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 01:00:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 01:00:34 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 15 01:02:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 01:02:52 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 15 01:05:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 01:05:17 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 15 01:07:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 01:07:50 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 15 01:10:47 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 01:10:47 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 01:12:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 01:12:56 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 15 01:15:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 01:15:39 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 15 01:17:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 01:17:57 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Jul 15 01:20:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 01:20:48 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 15 01:23:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 01:23:19 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Jul 15 01:25:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 01:25:43 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 15 01:28:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 01:28:03 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Jul 15 01:30:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 01:30:51 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 15 01:33:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 01:33:57 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 15 01:36:03 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 01:36:03 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 15 01:38:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 15 01:38:46 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Jul 15 01:41:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 01:41:01 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 01:44:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 01:44:48 fir-md1-s1 kernel: LustreError: Skipped 7 previous similar messages Jul 15 01:46:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 01:46:22 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 15 01:50:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 01:50:49 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 01:51:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 01:51:09 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 15 01:54:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 01:54:53 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 15 01:56:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 01:56:23 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 15 01:56:34 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1dedc57800, cur 1563180994 expire 1563180844 last 1563180767 Jul 15 01:56:34 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 15 02:00:53 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 02:00:53 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Jul 15 02:01:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 02:01:33 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 02:05:09 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 02:05:09 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 15 02:06:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 02:06:29 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 15 02:12:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 02:12:26 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Jul 15 02:12:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 02:12:27 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 15 02:15:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 02:15:41 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Jul 15 02:16:29 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 02:16:29 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 15 02:22:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 02:22:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 02:22:33 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 02:22:33 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 15 02:26:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 02:26:16 fir-md1-s1 kernel: LustreError: Skipped 12 previous similar messages Jul 15 02:26:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 02:26:43 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 15 02:33:07 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 02:33:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 02:33:07 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 02:33:07 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 15 02:37:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 02:37:31 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 15 02:37:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 02:37:46 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Jul 15 02:43:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 02:43:28 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 02:43:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 15 02:43:55 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Jul 15 02:47:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 02:47:34 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 02:48:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 02:48:23 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 15 02:53:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 02:53:33 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 15 02:53:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 02:53:56 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 15 02:57:57 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 02:57:57 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 15 02:58:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 02:58:41 fir-md1-s1 kernel: LustreError: Skipped 15 previous similar messages Jul 15 03:03:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 03:03:45 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 15 03:05:19 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 03:05:19 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 15 03:08:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 03:08:03 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 15 03:13:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 03:13:55 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 15 03:15:22 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 03:15:22 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 15 03:15:49 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 03:15:49 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 15 03:18:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 03:18:11 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 15 03:24:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 03:24:00 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 15 03:26:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 03:26:50 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 15 03:27:02 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 03:27:02 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 03:28:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 03:28:11 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 15 03:34:00 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 03:34:00 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 03:37:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 03:37:02 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 15 03:37:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 03:37:40 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 15 03:38:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 03:38:16 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 15 03:44:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 03:44:01 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 15 03:47:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 03:47:55 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 15 03:48:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 03:48:17 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 15 03:48:53 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 03:48:53 fir-md1-s1 kernel: LustreError: Skipped 8 previous similar messages Jul 15 03:54:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 03:54:13 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 03:58:30 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 03:58:30 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 15 03:58:40 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 15 03:58:40 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 15 04:04:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 04:04:26 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 04:06:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 23a2ad6b-df40-bb2e-b9a6-1311fa9a1b7e (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f330f01ac00, cur 1563188795 expire 1563188645 last 1563188568 Jul 15 04:08:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 04:08:33 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 15 04:10:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 04:10:47 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 15 04:10:51 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 04:10:51 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 15 04:14:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 04:14:46 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 15 04:16:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 04:18:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 04:18:45 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 15 04:20:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 04:20:27 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 04:20:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 04:20:55 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 15 04:24:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 04:24:49 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 04:26:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 04:26:04 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 04:28:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 04:28:48 fir-md1-s1 kernel: Lustre: Skipped 102 previous similar messages Jul 15 04:31:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 04:31:01 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 15 04:35:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 04:35:01 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 15 04:36:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 04:36:48 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 15 04:39:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 04:39:04 fir-md1-s1 kernel: Lustre: Skipped 94 previous similar messages Jul 15 04:41:07 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 04:41:07 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 15 04:45:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 04:45:01 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 15 04:48:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 04:48:08 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Jul 15 04:49:06 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 04:49:06 fir-md1-s1 kernel: Lustre: Skipped 132 previous similar messages Jul 15 04:52:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 04:52:10 fir-md1-s1 kernel: Lustre: Skipped 82 previous similar messages Jul 15 04:55:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 04:55:27 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 15 04:59:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 04:59:16 fir-md1-s1 kernel: Lustre: Skipped 79 previous similar messages Jul 15 05:02:11 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 05:02:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 05:02:11 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 15 05:05:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 05:05:36 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 05:09:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 05:09:16 fir-md1-s1 kernel: Lustre: Skipped 109 previous similar messages Jul 15 05:13:36 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 05:13:36 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 15 05:16:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 05:16:12 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 05:19:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 05:19:24 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Jul 15 05:23:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2fe090e400, cur 1563193423 expire 1563193273 last 1563193196 Jul 15 05:23:43 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 15 05:23:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 05:23:57 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 05:26:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 05:26:58 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 05:29:32 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 05:29:32 fir-md1-s1 kernel: Lustre: Skipped 59 previous similar messages Jul 15 05:33:34 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 05:33:34 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 05:37:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 05:37:05 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 15 05:37:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 05:37:08 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 15 05:39:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 05:39:54 fir-md1-s1 kernel: Lustre: Skipped 95 previous similar messages Jul 15 05:40:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 05:40:20 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 05:46:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 05:47:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 05:47:10 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 15 05:47:11 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 05:47:11 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 15 05:50:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 05:50:05 fir-md1-s1 kernel: Lustre: Skipped 71 previous similar messages Jul 15 05:52:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 05:57:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 05:57:20 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 05:58:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 05:58:13 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 15 06:00:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 06:00:12 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 15 06:07:44 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 06:07:44 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 15 06:09:56 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 06:09:56 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 15 06:10:20 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 06:10:20 fir-md1-s1 kernel: Lustre: Skipped 97 previous similar messages Jul 15 06:16:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 06:16:16 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 06:18:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 06:18:15 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 15 06:20:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.30.23@o2ib6, removing former export from same NID Jul 15 06:20:04 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 15 06:20:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 06:20:26 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 15 06:26:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 06:28:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 06:28:58 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 15 06:30:05 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 06:30:05 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 15 06:30:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2ecfc1c400, cur 1563197409 expire 1563197259 last 1563197182 Jul 15 06:30:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 06:30:35 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 15 06:34:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 06:39:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 06:39:23 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 15 06:40:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 06:40:06 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Jul 15 06:40:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 06:40:36 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 15 06:41:33 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 06:41:33 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 06:49:24 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 06:49:24 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 15 06:50:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 06:50:17 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Jul 15 06:51:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 06:51:15 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 15 06:57:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 06:57:55 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 06:59:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 06:59:28 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 07:02:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 07:02:24 fir-md1-s1 kernel: Lustre: Skipped 65 previous similar messages Jul 15 07:05:50 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f4449627400, cur 1563199550 expire 1563199400 last 1563199323 Jul 15 07:06:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 07:06:03 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 15 07:07:57 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 07:07:57 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 15 07:09:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 07:09:45 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 15 07:12:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 07:12:53 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 15 07:16:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 07:16:42 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 15 07:18:40 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 07:18:40 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 15 07:20:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 07:20:54 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 15 07:23:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 07:23:02 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 15 07:26:46 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 07:26:46 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 15 07:31:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 07:31:09 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 07:33:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 07:33:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 07:33:04 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 07:33:04 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 15 07:37:47 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 07:37:47 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 15 07:41:12 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 07:41:12 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 15 07:43:51 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 07:43:51 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 15 07:47:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 07:47:50 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 15 07:49:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 07:49:41 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 15 07:51:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 07:51:26 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 15 07:54:08 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 07:54:08 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 15 07:58:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 07:58:23 fir-md1-s1 kernel: Lustre: Skipped 47 previous similar messages Jul 15 08:00:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 08:01:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 08:01:33 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Jul 15 08:04:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 08:04:20 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Jul 15 08:10:15 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 08:10:15 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 15 08:11:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 08:11:35 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Jul 15 08:14:08 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 08:14:08 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 08:14:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 08:14:32 fir-md1-s1 kernel: Lustre: Skipped 77 previous similar messages Jul 15 08:21:30 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 08:21:30 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 15 08:21:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 08:21:44 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 08:24:50 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 08:24:50 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 15 08:30:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 08:30:18 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 15 08:31:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 08:31:52 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 08:33:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 08:33:42 fir-md1-s1 kernel: Lustre: Skipped 66 previous similar messages Jul 15 08:34:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 08:34:56 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 15 08:35:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1de7e58000, cur 1563204925 expire 1563204775 last 1563204698 Jul 15 08:42:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 08:42:16 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Jul 15 08:45:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 08:45:14 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 15 08:45:55 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 08:45:55 fir-md1-s1 kernel: Lustre: Skipped 61 previous similar messages Jul 15 08:51:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 08:51:14 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 15 08:52:54 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 08:53:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 08:53:13 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 15 08:56:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 08:56:01 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 15 08:56:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 08:56:01 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 15 09:03:38 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 09:03:38 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Jul 15 09:05:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 09:06:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 09:06:02 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 15 09:06:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 09:06:02 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Jul 15 09:14:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 09:14:02 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 15 09:16:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 09:16:15 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 09:17:35 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 09:17:35 fir-md1-s1 kernel: Lustre: Skipped 119 previous similar messages Jul 15 09:18:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 09:18:29 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 15 09:24:25 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 09:24:25 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Jul 15 09:27:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 09:27:25 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 15 09:27:46 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 09:27:46 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 15 09:28:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 09:28:48 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 15 09:34:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 09:34:47 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 15 09:37:35 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 09:37:58 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 09:37:58 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 15 09:38:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 09:38:52 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Jul 15 09:44:53 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 09:44:53 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 15 09:48:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 09:48:09 fir-md1-s1 kernel: Lustre: Skipped 102 previous similar messages Jul 15 09:49:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 09:49:25 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 15 09:55:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 09:55:15 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Jul 15 09:58:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 09:58:13 fir-md1-s1 kernel: Lustre: Skipped 62 previous similar messages Jul 15 09:59:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 09:59:32 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 15 10:04:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 10:04:19 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 10:05:28 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 10:05:28 fir-md1-s1 kernel: Lustre: Skipped 30 previous similar messages Jul 15 10:05:55 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 10:05:55 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 10:08:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 10:08:24 fir-md1-s1 kernel: Lustre: Skipped 109 previous similar messages Jul 15 10:11:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 10:11:18 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 15 10:15:49 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 10:15:49 fir-md1-s1 kernel: Lustre: Skipped 22 previous similar messages Jul 15 10:18:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 10:18:43 fir-md1-s1 kernel: Lustre: Skipped 73 previous similar messages Jul 15 10:21:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 10:21:25 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 15 10:26:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 10:26:27 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 15 10:28:13 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3671f01000, cur 1563211693 expire 1563211543 last 1563211466 Jul 15 10:28:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 10:28:47 fir-md1-s1 kernel: Lustre: Skipped 68 previous similar messages Jul 15 10:33:06 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.22.20@o2ib6, removing former export from same NID Jul 15 10:33:06 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 10:36:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 10:36:31 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 10:38:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 10:38:44 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 10:38:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 10:38:51 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 15 10:41:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 10:42:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 10:43:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 10:43:09 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Jul 15 10:44:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 10:44:31 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 10:46:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 10:46:40 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 15 10:48:48 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 10:48:48 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 10:49:09 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 10:49:09 fir-md1-s1 kernel: Lustre: Skipped 111 previous similar messages Jul 15 10:53:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 15 10:53:33 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 15 10:55:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 10:55:41 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 10:56:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 10:56:42 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 15 10:59:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 10:59:11 fir-md1-s1 kernel: Lustre: Skipped 126 previous similar messages Jul 15 11:03:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 11:03:33 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 15 11:07:23 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 11:07:23 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 11:09:18 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 11:09:18 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Jul 15 11:09:35 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 11:09:35 fir-md1-s1 kernel: Lustre: Skipped 86 previous similar messages Jul 15 11:13:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 11:13:33 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 15 11:17:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 11:17:39 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 15 11:19:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 11:19:43 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Jul 15 11:24:02 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 11:24:02 fir-md1-s1 kernel: Lustre: Skipped 55 previous similar messages Jul 15 11:24:46 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 11:25:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client eef76c71-6455-3c9c-c2bd-e13c2b066def (at 10.8.30.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34f15d3c00, cur 1563215104 expire 1563214954 last 1563214877 Jul 15 11:27:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 11:27:52 fir-md1-s1 kernel: Lustre: Skipped 41 previous similar messages Jul 15 11:30:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 11:30:02 fir-md1-s1 kernel: Lustre: Skipped 141 previous similar messages Jul 15 11:34:03 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 11:34:03 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 15 11:35:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 11:35:19 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 11:37:55 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 11:37:55 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 15 11:40:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 11:40:06 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 15 11:42:59 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 5048051f-aacc-10b9-d9da-eb27fb049919 (at 10.9.104.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f24f8ad4800, cur 1563216179 expire 1563216029 last 1563215952 Jul 15 11:42:59 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Jul 15 11:44:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 11:44:04 fir-md1-s1 kernel: Lustre: Skipped 53 previous similar messages Jul 15 11:44:15 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 018b4088-9100-7f5b-2709-38dd7f461ac7 (at 10.8.8.29@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff8f2501a69400, cur 1563216255 expire 1563216105 last 1563216084 Jul 15 11:44:15 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 15 11:45:11 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 887b140e-8cff-f857-a016-9d4798eb3a24 (at 10.8.8.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1505cb2c00, cur 1563216311 expire 1563216161 last 1563216084 Jul 15 11:45:11 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 15 11:46:27 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 6e5f9da5-7e02-257b-d9c3-9ff6edd45e41 (at 10.9.104.25@o2ib4) in 180 seconds. I think it's dead, and I am evicting it. exp ffff8f364fef9400, cur 1563216387 expire 1563216237 last 1563216207 Jul 15 11:47:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 8a6403b0-19b9-9d96-c101-52e3001fff6c (at 10.9.104.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f3d0b231800, cur 1563216437 expire 1563216287 last 1563216210 Jul 15 11:48:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 11:48:02 fir-md1-s1 kernel: Lustre: Skipped 46 previous similar messages Jul 15 11:50:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 11:50:12 fir-md1-s1 kernel: Lustre: Skipped 115 previous similar messages Jul 15 11:54:13 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 11:54:13 fir-md1-s1 kernel: Lustre: Skipped 42 previous similar messages Jul 15 11:54:45 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 91e21a4a-f1ae-e50e-7e41-21aa1b29cf61 (at 10.9.113.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1cf1ce8800, cur 1563216885 expire 1563216735 last 1563216658 Jul 15 11:54:45 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 15 11:55:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 11:58:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client eb03d68c-4477-fd95-4120-c15d0364314e (at 10.8.22.20@o2ib6) reconnecting Jul 15 11:58:09 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 15 12:00:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 12:00:18 fir-md1-s1 kernel: Lustre: Skipped 101 previous similar messages Jul 15 12:02:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 33678df3-cbf6-7b66-f13e-728347cfb474 (at 10.9.113.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f34fa331000, cur 1563217341 expire 1563217191 last 1563217114 Jul 15 12:02:21 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 15 12:04:18 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 12:04:18 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Jul 15 12:07:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.30.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 12:07:31 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 12:08:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 12:08:16 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Jul 15 12:10:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 12:10:21 fir-md1-s1 kernel: Lustre: Skipped 69 previous similar messages Jul 15 12:14:31 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 12:14:31 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 15 12:18:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 12:18:17 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 15 12:20:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 12:20:21 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Jul 15 12:24:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 12:24:23 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 12:27:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 12:27:17 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Jul 15 12:28:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 12:28:19 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 15 12:30:34 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 12:30:34 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Jul 15 12:37:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 12:37:23 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Jul 15 12:38:39 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 12:38:39 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 15 12:40:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 12:40:36 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 15 12:42:38 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 12:47:23 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 12:47:23 fir-md1-s1 kernel: Lustre: Skipped 78 previous similar messages Jul 15 12:48:56 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 12:48:56 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 15 12:50:36 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 12:50:36 fir-md1-s1 kernel: Lustre: Skipped 113 previous similar messages Jul 15 12:56:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 12:56:50 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 12:58:10 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 12:58:10 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 15 12:59:13 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 12:59:13 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Jul 15 13:00:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 13:00:54 fir-md1-s1 kernel: Lustre: Skipped 105 previous similar messages Jul 15 13:05:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f15065bb800, cur 1563221136 expire 1563220986 last 1563220909 Jul 15 13:05:36 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Jul 15 13:08:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 13:08:04 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 15 13:08:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 13:08:33 fir-md1-s1 kernel: Lustre: Skipped 58 previous similar messages Jul 15 13:09:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 13:09:16 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 13:11:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 13:11:04 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 15 13:16:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f281ff36c00, cur 1563221794 expire 1563221644 last 1563221567 Jul 15 13:18:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 13:18:33 fir-md1-s1 kernel: Lustre: Skipped 76 previous similar messages Jul 15 13:19:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 13:19:17 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 15 13:19:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 13:19:38 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 13:21:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 13:21:21 fir-md1-s1 kernel: Lustre: Skipped 123 previous similar messages Jul 15 13:29:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 13:29:25 fir-md1-s1 kernel: Lustre: Skipped 48 previous similar messages Jul 15 13:29:41 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 13:29:41 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 15 13:30:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 13:30:23 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 13:31:27 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 13:31:27 fir-md1-s1 kernel: Lustre: Skipped 85 previous similar messages Jul 15 13:39:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 13:39:44 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Jul 15 13:40:21 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.12.12@o2ib6, removing former export from same NID Jul 15 13:40:21 fir-md1-s1 kernel: Lustre: Skipped 63 previous similar messages Jul 15 13:41:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 13:41:31 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 15 13:45:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 13:45:42 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 13:49:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bf3478cc-569b-5c14-1a71-20ca1e1f08aa (at 10.8.12.12@o2ib6) reconnecting Jul 15 13:49:49 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Jul 15 13:51:42 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 13:51:42 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Jul 15 13:51:48 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 13:51:48 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Jul 15 13:55:52 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 13:55:52 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 14:00:18 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 14:00:18 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Jul 15 14:01:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 1b7aecc8-a455-dad2-efc3-59dffd90c0d4 (at 10.8.12.12@o2ib6) Jul 15 14:01:50 fir-md1-s1 kernel: Lustre: Skipped 116 previous similar messages Jul 15 14:02:25 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 14:02:25 fir-md1-s1 kernel: Lustre: Skipped 75 previous similar messages Jul 15 14:10:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 14:10:26 fir-md1-s1 kernel: Lustre: Skipped 33 previous similar messages Jul 15 14:11:54 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to b270ac6a-a3a3-60fc-aec3-c49e072fb0ae (at 10.8.30.23@o2ib6) Jul 15 14:11:54 fir-md1-s1 kernel: Lustre: Skipped 74 previous similar messages Jul 15 14:12:41 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 14:12:41 fir-md1-s1 kernel: Lustre: Skipped 39 previous similar messages Jul 15 14:20:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 14:20:32 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Jul 15 14:20:52 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 6dc651d0-2b7a-dd35-f234-bffd4712bc50 (at 10.8.30.23@o2ib6) reconnecting Jul 15 14:20:52 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 15 14:22:21 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 14:22:21 fir-md1-s1 kernel: Lustre: Skipped 100 previous similar messages Jul 15 14:22:50 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 14:22:50 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Jul 15 14:30:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 14:30:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 14:30:55 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Jul 15 14:32:27 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 14:32:27 fir-md1-s1 kernel: Lustre: Skipped 89 previous similar messages Jul 15 14:34:24 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 14:34:24 fir-md1-s1 kernel: Lustre: Skipped 60 previous similar messages Jul 15 14:39:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 14:39:31 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Jul 15 14:41:03 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 14:41:03 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 14:42:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 14:42:31 fir-md1-s1 kernel: Lustre: Skipped 98 previous similar messages Jul 15 14:46:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 14:46:16 fir-md1-s1 kernel: Lustre: Skipped 57 previous similar messages Jul 15 14:50:34 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 14:50:34 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 14:51:17 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 14:51:17 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Jul 15 14:52:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7d2b05ed-81cd-3d62-bb9f-e3f301bfd456 (at 10.8.11.6@o2ib6) Jul 15 14:52:31 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Jul 15 14:56:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 14:56:16 fir-md1-s1 kernel: Lustre: Skipped 37 previous similar messages Jul 15 15:01:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 64093eed-1899-7457-95e6-ff7526581ffb (at 10.8.10.21@o2ib6) reconnecting Jul 15 15:01:21 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 15 15:02:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 15:02:43 fir-md1-s1 kernel: Lustre: Skipped 87 previous similar messages Jul 15 15:04:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.12.12@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 15:04:12 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Jul 15 15:07:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 15:07:17 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 15:11:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 17e26c1e-4877-4fff-89e1-78bf5463918b (at 10.8.11.6@o2ib6) reconnecting Jul 15 15:11:33 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Jul 15 15:13:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 02d0382c-ac98-fa2d-e4db-d0092db77da5 (at 10.8.22.20@o2ib6) Jul 15 15:13:01 fir-md1-s1 kernel: Lustre: Skipped 54 previous similar messages Jul 15 15:17:29 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.11.6@o2ib6, removing former export from same NID Jul 15 15:17:29 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Jul 15 15:18:38 fir-md1-s1 kernel: Lustre: 23697:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1563229111/real 0] req@ffff8f360672dd00 x1636733705317840/t0(0) o104->fir-MDT0002@10.8.7.31@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1563229118 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jul 15 15:18:40 fir-md1-s1 kernel: LustreError: 46522:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f34c8aa5850 x1638236447926464/t0(0) o4->e53089e0-0379-2982-632f-afbd57f75e4f@10.8.2.32@o2ib6:23/0 lens 504/448 e 1 to 0 dl 1563229133 ref 1 fl Interpret:/0/0 rc 0/0 Jul 15 15:18:41 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1563229114/real 0] req@ffff8f0db4e5ad00 x1636733705353616/t0(0) o106->fir-MDT0000@10.8.29.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1563229121 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jul 15 15:18:41 fir-md1-s1 kernel: Lustre: 23598:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 15 15:18:42 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:18:43 fir-md1-s1 kernel: Lustre: 20463:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1563229116/real 0] req@ffff8f3177b75400 x1636733705365360/t0(0) o104->fir-MDT0000@10.8.28.2@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1563229123 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jul 15 15:18:43 fir-md1-s1 kernel: Lustre: 20463:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 33 previous similar messages Jul 15 15:18:43 fir-md1-s1 kernel: LNetError: 20189:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:18:43 fir-md1-s1 kernel: LustreError: 27602:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f184b563850 x1638886645534656/t0(0) o4->666b60d6-ed92-c98b-c78c-4bfc3f3e7231@10.8.16.2@o2ib6:23/0 lens 504/448 e 1 to 0 dl 1563229133 ref 1 fl Interpret:/0/0 rc 0/0 Jul 15 15:18:43 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f33d8ec6000 Jul 15 15:18:43 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with e53089e0-0379-2982-632f-afbd57f75e4f (at 10.8.2.32@o2ib6), client will retry: rc = -110 Jul 15 15:18:43 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Jul 15 15:18:44 fir-md1-s1 kernel: LustreError: 20189:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f1bb0529c00 Jul 15 15:18:45 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:18:45 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Jul 15 15:18:45 fir-md1-s1 kernel: Lustre: 23714:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2a223a8600 x1638276954421856/t352605293928(0) o36->ef0748a0-58bc-3624-ed96-74860cd1e591@10.8.0.66@o2ib6:20/0 lens 504/2888 e 1 to 0 dl 1563229130 ref 2 fl Interpret:/0/0 rc 0/0 Jul 15 15:18:47 fir-md1-s1 kernel: Lustre: 23733:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f2a223af800 x1637883407800928/t0(0) o101->aa3ee41d-cac0-6749-5220-bb62e9eebc36@10.8.28.5@o2ib6:21/0 lens 576/3264 e 1 to 0 dl 1563229131 ref 2 fl Interpret:/0/0 rc 0/0 Jul 15 15:18:48 fir-md1-s1 kernel: LNetError: 20188:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:18:49 fir-md1-s1 kernel: Lustre: 23615:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f056c851200 x1631542709475504/t352605286701(0) o36->903c51ef-2159-9907-073d-897a3f432dcf@10.9.109.11@o2ib4:24/0 lens 488/3152 e 1 to 0 dl 1563229134 ref 2 fl Interpret:/0/0 rc 0/0 Jul 15 15:18:49 fir-md1-s1 kernel: Lustre: 23615:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jul 15 15:18:51 fir-md1-s1 kernel: Lustre: 23615:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f0db4e58000 x1635204891654560/t0(0) o101->f6ea22f6-446c-b33a-7f85-ddd4280dae8d@10.9.101.23@o2ib4:26/0 lens 576/3264 e 1 to 0 dl 1563229136 ref 2 fl Interpret:/0/0 rc 0/0 Jul 15 15:18:51 fir-md1-s1 kernel: Lustre: 23615:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 36 previous similar messages Jul 15 15:18:52 fir-md1-s1 kernel: Lustre: 20996:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:1s); client may timeout. req@ffff8f2a223af800 x1637883407800928/t0(0) o101->aa3ee41d-cac0-6749-5220-bb62e9eebc36@10.8.28.5@o2ib6:21/0 lens 576/536 e 1 to 0 dl 1563229131 ref 1 fl Complete:/0/0 rc 0/0 Jul 15 15:18:52 fir-md1-s1 kernel: Lustre: 20996:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 53848 previous similar messages Jul 15 15:18:53 fir-md1-s1 kernel: Lustre: 20463:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563229126/real 1563229126] req@ffff8f2de4b91e00 x1636733705377328/t0(0) o104->fir-MDT0000@10.8.10.36@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1563229133 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Jul 15 15:18:53 fir-md1-s1 kernel: Lustre: 20463:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 488 previous similar messages Jul 15 15:18:55 fir-md1-s1 kernel: Lustre: 20738:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f08f1135d00 x1638894534705040/t0(0) o101->70f17c05-8e9e-e3e3-0fb3-adadf2c8b10a@10.9.103.22@o2ib4:0/0 lens 480/0 e 1 to 0 dl 1563229140 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 15 15:18:55 fir-md1-s1 kernel: Lustre: 20738:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 357 previous similar messages Jul 15 15:19:01 fir-md1-s1 kernel: Lustre: 23598:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:2s); client may timeout. req@ffff8f08440f3600 x1634161738022496/t0(0) o101->32315fe6-6915-bd82-691a-5460d13ab6db@10.9.103.27@o2ib4:29/0 lens 480/0 e 1 to 0 dl 1563229139 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 15 15:19:02 fir-md1-s1 kernel: Lustre: 23607:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:12s); client may timeout. req@ffff8f2a223a8600 x1638276954421856/t352605293928(0) o36->ef0748a0-58bc-3624-ed96-74860cd1e591@10.8.0.66@o2ib6:20/0 lens 504/424 e 1 to 0 dl 1563229130 ref 1 fl Complete:/0/0 rc 0/0 Jul 15 15:19:03 fir-md1-s1 kernel: Lustre: 23615:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f08f1131500 x1631742464335136/t0(0) o101->9101e47c-5087-9ebf-bb20-6ff2bf817bf0@10.9.101.32@o2ib4:8/0 lens 576/0 e 1 to 0 dl 1563229148 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 15 15:19:03 fir-md1-s1 kernel: Lustre: 23615:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 993 previous similar messages Jul 15 15:19:08 fir-md1-s1 kernel: LustreError: 46524:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8f184b564c50 x1637882170889552/t0(0) o4->f7faac5e-5757-f826-f11b-7d0a6430dabe@10.8.8.27@o2ib6:14/0 lens 488/448 e 1 to 0 dl 1563229154 ref 1 fl Interpret:/0/0 rc 0/0 Jul 15 15:19:09 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:19:09 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Jul 15 15:19:09 fir-md1-s1 kernel: LustreError: 20187:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f24e360d200 Jul 15 15:19:09 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with f7faac5e-5757-f826-f11b-7d0a6430dabe (at 10.8.8.27@o2ib6), client will retry: rc = -110 Jul 15 15:19:09 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 15 15:19:14 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 40s: evicting client at 10.8.8.18@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f09644198c0/0x5d9ee640c35e808a lrc: 3/0,0 mode: PR/PR res: [0x2c002c4ce:0x13d1f:0x0].0x0 bits 0x1b/0x0 rrc: 34 type: IBT flags: 0x60200400000020 nid: 10.8.8.18@o2ib6 remote: 0xbd8ddbfa7a81dce2 expref: 15305 pid: 23685 timeout: 2344214 lvb_type: 0 Jul 15 15:19:14 fir-md1-s1 kernel: Lustre: 23691:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:15s); client may timeout. req@ffff8f08ff828000 x1631567180206448/t0(0) o101->35fe08e4-c10b-c2c7-284d-8125b5106002@10.9.107.3@o2ib4:29/0 lens 576/0 e 1 to 0 dl 1563229139 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 15 15:19:14 fir-md1-s1 kernel: LustreError: 21452:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.22.34@o2ib6: deadline 30:5s ago req@ffff8f28d63ea100 x1631646493744000/t0(0) o101->f03aa5e8-f764-2262-c217-2e99830bfe5f@10.8.22.34@o2ib6:9/0 lens 576/0 e 0 to 0 dl 1563229149 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 15 15:19:14 fir-md1-s1 kernel: LustreError: 21452:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 36 previous similar messages Jul 15 15:19:15 fir-md1-s1 kernel: Lustre: 23691:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 122 previous similar messages Jul 15 15:19:15 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 40s: evicting client at 10.8.22.33@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f20ade33a80/0x5d9ee640c379ca6e lrc: 3/0,0 mode: PR/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 901 type: IBT flags: 0x60200400000020 nid: 10.8.22.33@o2ib6 remote: 0xc3eed59f75023b34 expref: 8 pid: 97642 timeout: 2344215 lvb_type: 0 Jul 15 15:19:17 fir-md1-s1 kernel: LustreError: 22285:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.7.27@o2ib6: deadline 30:8s ago req@ffff8f16c9304500 x1631578064405024/t0(0) o101->9b7917ef-4055-daa1-69c4-53b2ed51bc97@10.8.7.27@o2ib6:9/0 lens 584/0 e 0 to 0 dl 1563229149 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 15 15:19:17 fir-md1-s1 kernel: LustreError: 22285:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 17 previous similar messages Jul 15 15:19:19 fir-md1-s1 kernel: Lustre: 20738:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f06ac52ad00 x1634178006941776/t0(0) o101->d82be57b-2f2b-1591-b61e-7d36849f0064@10.9.109.71@o2ib4:24/0 lens 576/0 e 1 to 0 dl 1563229164 ref 2 fl New:/0/ffffffff rc 0/-1 Jul 15 15:19:19 fir-md1-s1 kernel: Lustre: 20738:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2114 previous similar messages Jul 15 15:19:43 fir-md1-s1 kernel: LustreError: 21987:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 20+0s req@ffff8f2811aa0c50 x1637882170889552/t0(0) o4->f7faac5e-5757-f826-f11b-7d0a6430dabe@10.8.8.27@o2ib6:13/0 lens 488/448 e 1 to 0 dl 1563229183 ref 1 fl Interpret:/2/0 rc 0/0 Jul 15 15:19:43 fir-md1-s1 kernel: LustreError: 21987:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 15 previous similar messages Jul 15 15:19:51 fir-md1-s1 kernel: Lustre: 23615:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f056c856900 x1638088205671680/t0(0) o101->9901f7bd-3861-a1cb-77e0-01bd9d079c38@10.9.110.3@o2ib4:26/0 lens 576/0 e 0 to 0 dl 1563229196 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 15 15:19:51 fir-md1-s1 kernel: Lustre: 23615:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2481 previous similar messages Jul 15 15:20:01 fir-md1-s1 kernel: LustreError: 23671:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229111, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f320ee8b840/0x5d9ee640c378105d lrc: 3/0,1 mode: --/CW res: [0x2c002c39f:0x28a7:0x0].0x0 bits 0x2/0x0 rrc: 501 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 23671 timeout: 0 lvb_type: 0 Jul 15 15:20:01 fir-md1-s1 kernel: LustreError: 23671:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 15 15:20:05 fir-md1-s1 kernel: LustreError: 23704:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229115, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8f320e134ec0/0x5d9ee640c37ae213 lrc: 3/1,0 mode: --/PR res: [0x2c002c39f:0x28a7:0x0].0x0 bits 0x13/0x0 rrc: 501 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23704 timeout: 0 lvb_type: 0 Jul 15 15:20:05 fir-md1-s1 kernel: LustreError: 23704:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 4 previous similar messages Jul 15 15:20:05 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:20:05 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Jul 15 15:20:05 fir-md1-s1 kernel: LustreError: 20190:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8f2889f84600 Jul 15 15:20:05 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO write error with f7faac5e-5757-f826-f11b-7d0a6430dabe (at 10.8.8.27@o2ib6), client will retry: rc = -110 Jul 15 15:20:05 fir-md1-s1 kernel: Lustre: 21987:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:22s); client may timeout. req@ffff8f2811aa0c50 x1637882170889552/t0(0) o4->f7faac5e-5757-f826-f11b-7d0a6430dabe@10.8.8.27@o2ib6:13/0 lens 488/448 e 1 to 0 dl 1563229183 ref 1 fl Complete:/2/ffffffff rc -110/-1 Jul 15 15:20:05 fir-md1-s1 kernel: Lustre: 21987:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 635 previous similar messages Jul 15 15:20:06 fir-md1-s1 kernel: LustreError: 23665:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229116, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2c943a1f80/0x5d9ee640c37bcff0 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 904 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23665 timeout: 0 lvb_type: 0 Jul 15 15:20:06 fir-md1-s1 kernel: LustreError: 23665:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 91 previous similar messages Jul 15 15:20:08 fir-md1-s1 kernel: LustreError: 21414:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229118, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f12dfa37500/0x5d9ee640c37d6c1e lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 904 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 21414 timeout: 0 lvb_type: 0 Jul 15 15:20:08 fir-md1-s1 kernel: LustreError: 21414:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 242 previous similar messages Jul 15 15:20:22 fir-md1-s1 kernel: LustreError: 23697:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229132, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2815b21200/0x5d9ee640c37db5c8 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 904 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23697 timeout: 0 lvb_type: 0 Jul 15 15:20:22 fir-md1-s1 kernel: LustreError: 23697:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 86 previous similar messages Jul 15 15:20:31 fir-md1-s1 kernel: LustreError: 10502:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229141, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2f05fc7500/0x5d9ee640c37db6c4 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 904 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10502 timeout: 0 lvb_type: 0 Jul 15 15:20:31 fir-md1-s1 kernel: LustreError: 10502:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 15 15:20:47 fir-md1-s1 kernel: LustreError: 23672:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229157, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f143c6f98c0/0x5d9ee640c37db845 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 904 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23672 timeout: 0 lvb_type: 0 Jul 15 15:20:47 fir-md1-s1 kernel: LustreError: 23672:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 4 previous similar messages Jul 15 15:20:55 fir-md1-s1 kernel: Lustre: 20738:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f12300a8600 x1631565736890448/t0(0) o101->42800284-789e-e9cc-0ebd-dbacb154f6ac@10.9.107.31@o2ib4:0/0 lens 576/0 e 0 to 0 dl 1563229260 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 15 15:20:55 fir-md1-s1 kernel: Lustre: 20738:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5412 previous similar messages Jul 15 15:21:15 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 143s: evicting client at 10.8.28.5@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f33bdfd0900/0x5d9ee640c377f265 lrc: 3/0,0 mode: PR/PR res: [0x2c002c39f:0x28a7:0x0].0x0 bits 0x13/0x0 rrc: 500 type: IBT flags: 0x60200400000020 nid: 10.8.28.5@o2ib6 remote: 0x83a6390e06f652ed expref: 663 pid: 20996 timeout: 2344221 lvb_type: 0 Jul 15 15:21:16 fir-md1-s1 kernel: Lustre: 21428:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:143s); client may timeout. req@ffff8f162cf57800 x1638084117200976/t352605315585(0) o101->905c028c-e587-96e1-52d7-ae94e0d5428f@10.8.7.31@o2ib6:22/0 lens 1792/1192 e 1 to 0 dl 1563229132 ref 1 fl Complete:/0/0 rc 0/0 Jul 15 15:21:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.10.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 15:21:16 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Jul 15 15:21:16 fir-md1-s1 kernel: LustreError: 23704:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.104.63@o2ib4: deadline 30:1s ago req@ffff8f2e57d65100 x1633881576987616/t0(0) o101->ec935c16-6a63-f875-145b-2db5feba3892@10.9.104.63@o2ib4:14/0 lens 576/0 e 0 to 0 dl 1563229274 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jul 15 15:21:16 fir-md1-s1 kernel: LustreError: 23704:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 5 previous similar messages Jul 15 15:21:16 fir-md1-s1 kernel: Lustre: 21428:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 5463 previous similar messages Jul 15 15:21:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client 2cc0bc1b-7a1f-9dab-b36c-c6206a02385d (at 10.8.20.20@o2ib6) reconnecting Jul 15 15:21:33 fir-md1-s1 kernel: Lustre: Skipped 7802 previous similar messages Jul 15 15:21:56 fir-md1-s1 kernel: LNet: Service thread pid 21416 was inactive for 200.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 15 15:21:56 fir-md1-s1 kernel: Pid: 21416, comm: mdt00_015 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 15 15:21:56 fir-md1-s1 kernel: Call Trace: Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 15 15:21:56 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 15 15:21:56 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 15 15:21:56 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229316.21416 Jul 15 15:21:56 fir-md1-s1 kernel: Pid: 21368, comm: mdt00_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 15 15:21:56 fir-md1-s1 kernel: Call Trace: Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 15 15:21:56 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 15 15:21:56 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 15 15:21:56 fir-md1-s1 kernel: LNet: Service thread pid 97657 was inactive for 200.50s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 15 15:21:56 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Jul 15 15:21:56 fir-md1-s1 kernel: Pid: 97657, comm: mdt01_096 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 15 15:21:56 fir-md1-s1 kernel: Call Trace: Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 15 15:21:56 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 15 15:21:56 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 15 15:21:56 fir-md1-s1 kernel: Pid: 21369, comm: mdt00_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 15 15:21:56 fir-md1-s1 kernel: Call Trace: Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 15 15:21:56 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 15 15:21:56 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 15 15:21:56 fir-md1-s1 kernel: Pid: 97669, comm: mdt01_108 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 15 15:21:56 fir-md1-s1 kernel: Call Trace: Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 15 15:21:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 15 15:21:57 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 15 15:21:57 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 15 15:21:57 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Jul 15 15:21:57 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Jul 15 15:21:57 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Jul 15 15:21:57 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Jul 15 15:21:57 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Jul 15 15:21:57 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Jul 15 15:21:57 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 15 15:21:57 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 15 15:21:57 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 15 15:21:57 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 15 15:21:57 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 15 15:21:57 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 15 15:21:57 fir-md1-s1 kernel: LNet: Service thread pid 23560 was inactive for 200.82s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 15 15:21:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229317.10364 Jul 15 15:21:57 fir-md1-s1 kernel: LNet: Service thread pid 23749 was inactive for 200.47s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 15 15:21:57 fir-md1-s1 kernel: LNet: Skipped 182 previous similar messages Jul 15 15:21:58 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229318.23455 Jul 15 15:21:58 fir-md1-s1 kernel: LNet: Service thread pid 21460 was inactive for 200.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 15 15:21:58 fir-md1-s1 kernel: LNet: Skipped 130 previous similar messages Jul 15 15:21:59 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229319.23567 Jul 15 15:22:00 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229320.23717 Jul 15 15:22:12 fir-md1-s1 kernel: LNet: Service thread pid 20996 was inactive for 200.39s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 15 15:22:12 fir-md1-s1 kernel: LNet: Skipped 104 previous similar messages Jul 15 15:22:12 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229332.20996 Jul 15 15:22:21 fir-md1-s1 kernel: LNet: Service thread pid 23598 was inactive for 200.33s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 15 15:22:21 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Jul 15 15:22:21 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229341.23598 Jul 15 15:22:22 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229342.23607 Jul 15 15:22:34 fir-md1-s1 kernel: LNetError: 20190:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:22:35 fir-md1-s1 kernel: LNet: Service thread pid 21452 was inactive for 200.13s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 15 15:22:35 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Jul 15 15:22:35 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229355.21452 Jul 15 15:22:38 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229358.23672 Jul 15 15:22:46 fir-md1-s1 kernel: LustreError: 23733:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229275, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f2d8273cec0/0x5d9ee640c37de992 lrc: 3/1,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 881 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 23733 timeout: 0 lvb_type: 0 Jul 15 15:22:46 fir-md1-s1 kernel: LustreError: 23733:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 4 previous similar messages Jul 15 15:23:01 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to 2903d57c-5762-79f0-1085-17dddf3a1579 (at 10.8.23.12@o2ib6) Jul 15 15:23:01 fir-md1-s1 kernel: Lustre: Skipped 15362 previous similar messages Jul 15 15:23:03 fir-md1-s1 kernel: Lustre: 20730:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f21538bb000 x1638894101777744/t0(0) o101->841377fb-5d3e-8b58-50de-caee09553c02@10.9.112.8@o2ib4:8/0 lens 576/0 e 0 to 0 dl 1563229388 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 15 15:23:03 fir-md1-s1 kernel: Lustre: 20730:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 11121 previous similar messages Jul 15 15:23:08 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:23:08 fir-md1-s1 kernel: LNetError: 20187:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Jul 15 15:24:36 fir-md1-s1 kernel: LNet: Service thread pid 20732 was inactive for 200.43s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 15 15:24:36 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Jul 15 15:24:36 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563229476.20732 Jul 15 15:25:02 fir-md1-s1 kernel: Lustre: fir-MDT0002: haven't heard from client 26f27ada-08f0-595f-95a1-db8559ff813e (at 10.8.8.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1ff6647400, cur 1563229502 expire 1563229352 last 1563229275 Jul 15 15:27:19 fir-md1-s1 kernel: Lustre: 23729:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8f3760e6ce00 x1634126899674128/t0(0) o101->a7aad8e9-6055-f520-5dcf-5ea6b8e2ae73@10.9.104.52@o2ib4:24/0 lens 576/0 e 0 to 0 dl 1563229644 ref 2 fl New:/2/ffffffff rc 0/-1 Jul 15 15:27:19 fir-md1-s1 kernel: Lustre: 23729:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 45820 previous similar messages Jul 15 15:29:27 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.10.21@o2ib6, removing former export from same NID Jul 15 15:29:27 fir-md1-s1 kernel: Lustre: Skipped 4240 previous similar messages Jul 15 15:30:24 fir-md1-s1 kernel: LNet: Service thread pid 23733 completed after 548.15s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jul 15 15:30:24 fir-md1-s1 kernel: Lustre: 97649:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:558s); client may timeout. req@ffff8f1e71560300 x1631545009821280/t0(0) o101->f5f74966-59a2-6619-dc33-28e321e9f975@10.9.108.31@o2ib4:6/0 lens 576/0 e 0 to 0 dl 1563229266 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Jul 15 15:30:24 fir-md1-s1 kernel: Lustre: 97649:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 42 previous similar messages Jul 15 15:30:24 fir-md1-s1 kernel: LustreError: 20460:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.15.6@o2ib6: deadline 100:448s ago req@ffff8f17072d6300 x1639150855008368/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1563229376 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 15 15:30:24 fir-md1-s1 kernel: LustreError: 20460:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 5 previous similar messages Jul 15 15:30:24 fir-md1-s1 kernel: LNet: Skipped 421 previous similar messages Jul 15 15:30:24 fir-md1-s1 kernel: LNetError: 20194:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) Jul 15 15:30:24 fir-md1-s1 kernel: LustreError: 97661:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f1bb05a4800 x1636733705968336/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 15 15:30:24 fir-md1-s1 kernel: LustreError: 97661:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jul 15 15:30:34 fir-md1-s1 kernel: Lustre: 23627:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563229824/real 1563229824] req@ffff8f2fd47b0f00 x1636733705955568/t0(0) o104->fir-MDT0002@10.8.17.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1563229834 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jul 15 15:30:44 fir-md1-s1 kernel: Lustre: 23627:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563229834/real 1563229834] req@ffff8f2fd47b0f00 x1636733705955568/t0(0) o104->fir-MDT0002@10.8.17.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1563229844 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 15 15:30:44 fir-md1-s1 kernel: Lustre: 23627:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 15 15:30:53 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f362c63da00/0x5d9ee640ba1257a2 lrc: 3/0,0 mode: PR/PR res: [0x2000222f5:0x2c5:0x0].0x0 bits 0x5b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x243de7f265dcd7a8 expref: 244418 pid: 23608 timeout: 2344913 lvb_type: 0 Jul 15 15:30:53 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 39 previous similar messages Jul 15 15:30:54 fir-md1-s1 kernel: Lustre: 23627:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563229844/real 1563229844] req@ffff8f2fd47b0f00 x1636733705955568/t0(0) o104->fir-MDT0002@10.8.17.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1563229854 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 15 15:30:54 fir-md1-s1 kernel: Lustre: 23627:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 15 15:31:04 fir-md1-s1 kernel: Lustre: 23627:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1563229854/real 1563229854] req@ffff8f2fd47b0f00 x1636733705955568/t0(0) o104->fir-MDT0002@10.8.17.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1563229864 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Jul 15 15:31:04 fir-md1-s1 kernel: Lustre: 23627:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 15 15:31:04 fir-md1-s1 kernel: LustreError: 23627:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.17.7@o2ib6) failed to reply to blocking AST (req@ffff8f2fd47b0f00 x1636733705955568 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8f15bbdad7c0/0x5d9ee640c377d363 lrc: 4/0,0 mode: PR/PR res: [0x2c002c39f:0x28a8:0x0].0x0 bits 0x13/0x0 rrc: 1090 type: IBT flags: 0x60200400000020 nid: 10.8.17.7@o2ib6 remote: 0x995d44ac889bc5d7 expref: 287 pid: 24576 timeout: 2344943 lvb_type: 0 Jul 15 15:31:04 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.17.7@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Jul 15 15:31:04 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 40s: evicting client at 10.8.17.7@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8f15bbdad7c0/0x5d9ee640c377d363 lrc: 3/0,0 mode: PR/PR res: [0x2c002c39f:0x28a8:0x0].0x0 bits 0x13/0x0 rrc: 1091 type: IBT flags: 0x60200400000020 nid: 10.8.17.7@o2ib6 remote: 0x995d44ac889bc5d7 expref: 288 pid: 24576 timeout: 0 lvb_type: 0 Jul 15 15:31:04 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Jul 15 15:31:04 fir-md1-s1 kernel: Lustre: 20996:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:4s); client may timeout. req@ffff8f324d363600 x1631621243475024/t0(0) o101->7904decb-1129-4831-4db2-1394d4834a08@10.9.108.47@o2ib4:0/0 lens 1768/0 e 0 to 0 dl 1563229860 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 15 15:31:04 fir-md1-s1 kernel: LustreError: 21677:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.1.23@o2ib6: deadline 30:4s ago req@ffff8f2fbdf80c00 x1635095650204480/t0(0) o101->02f653ee-3954-8dc8-cd3c-07c80d9ed9d2@10.8.1.23@o2ib6:0/0 lens 576/0 e 0 to 0 dl 1563229860 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jul 15 15:31:04 fir-md1-s1 kernel: LustreError: 21677:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 372 previous similar messages Jul 15 15:31:04 fir-md1-s1 kernel: LustreError: 23603:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f316db67200 x1636733706731744/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 15 15:31:04 fir-md1-s1 kernel: LustreError: 23603:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Jul 15 15:31:04 fir-md1-s1 kernel: Lustre: 20996:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 91946 previous similar messages Jul 15 15:31:06 fir-md1-s1 kernel: LustreError: 23584:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f2ec5a66600 x1636733706786368/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 15 15:31:06 fir-md1-s1 kernel: LustreError: 23584:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Jul 15 15:31:08 fir-md1-s1 kernel: LustreError: 97657:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f15b7e6a700 x1636733706844992/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 15 15:31:08 fir-md1-s1 kernel: LustreError: 97657:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Jul 15 15:31:26 fir-md1-s1 kernel: LustreError: 21676:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f2b17220900 x1636733707267456/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 15 15:31:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6bb1b23c-28f8-153d-8cc1-2ff0115f9167 (at 10.9.106.58@o2ib4) reconnecting Jul 15 15:31:33 fir-md1-s1 kernel: Lustre: Skipped 25644 previous similar messages Jul 15 15:31:33 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f240b67d100/0x5d9ee640a619cbb4 lrc: 3/0,0 mode: PR/PR res: [0x20000fb8f:0x672:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x243de7f25f98072b expref: 198956 pid: 23750 timeout: 2344953 lvb_type: 0 Jul 15 15:31:54 fir-md1-s1 kernel: LustreError: 24580:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1563229824, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8f227de20240/0x5d9ee640c39383bc lrc: 3/0,1 mode: --/PW res: [0x2000222f5:0x2c5:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 24580 timeout: 0 lvb_type: 0 Jul 15 15:31:54 fir-md1-s1 kernel: LustreError: 24580:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 15 15:31:56 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f3208797500/0x5d9ee640b61ac6a8 lrc: 3/0,0 mode: PR/PR res: [0x2000297f6:0x88a:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x243de7f26418bf6f expref: 173376 pid: 21379 timeout: 2344976 lvb_type: 0 Jul 15 15:31:56 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 8 previous similar messages Jul 15 15:33:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 7e64fcba-461c-e286-f780-b934c678bb43 (at 10.8.10.21@o2ib6) Jul 15 15:33:05 fir-md1-s1 kernel: Lustre: Skipped 22433 previous similar messages Jul 15 15:33:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.11.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Jul 15 15:33:23 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 6afac91a-e1c8-0ca6-0677-8b79f37ef46e (at 10.8.17.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f1675e89800, cur 1563230003 expire 1563229853 last 1563229776 Jul 15 15:33:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Jul 15 15:33:57 fir-md1-s1 kernel: LustreError: 24580:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8f16a45d3000 x1636733713971856/t0(0) o104->fir-MDT0000@10.8.9.9@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Jul 15 15:33:57 fir-md1-s1 kernel: LustreError: 24580:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 10 previous similar messages Jul 15 15:34:09 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Mon Jul 15 15:34:09 2019 Jul 15 15:34:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 2ade0b9c-5691-7fbe-1d3a-8c6ce8591788 (at 10.8.17.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8f2ee4895400, cur 1563230051 expire 1563229901 last 1563229824 Jul 15 15:34:11 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Mon Jul 15 15:34:11 2019 Jul 15 15:34:26 fir-md1-s1 kernel: LNet: Service thread pid 23077 was inactive for 200.25s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 15 15:34:26 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Jul 15 15:34:26 fir-md1-s1 kernel: Pid: 23077, comm: mdt02_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Jul 15 15:34:26 fir-md1-s1 kernel: Call Trace: Jul 15 15:34:26 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x890 [ptlrpc] Jul 15 15:34:26 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] Jul 15 15:34:26 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Jul 15 15:34:26 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x3e0 [mdt] Jul 15 15:34:26 fir-md1-s1 kernel: [] mdt_reint_object_lock+0x2c/0x60 [mdt] Jul 15 15:34:26 fir-md1-s1 kernel: [] mdt_reint_striped_lock+0x8c/0x510 [mdt] Jul 15 15:34:26 fir-md1-s1 kernel: [] mdt_reint_setattr+0x6c8/0x1340 [mdt] Jul 15 15:34:26 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Jul 15 15:34:26 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Jul 15 15:34:26 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Jul 15 15:34:26 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Jul 15 15:34:26 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Jul 15 15:34:26 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Jul 15 15:34:26 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Jul 15 15:34:26 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Jul 15 15:34:26 fir-md1-s1 kernel: [] 0xffffffffffffffff Jul 15 15:34:26 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1563230066.23077 Jul 15 15:34:26 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.9.9@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8f31c0f9e540/0x5d9ee640b61b6f03 lrc: 3/0,0 mode: PR/PR res: [0x2000297f6:0x882:0x0].0x0 bits 0x5b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.9.9@o2ib6 remote: 0x243de7f26418e7fc expref: 16849 pid: 25680 timeout: 2345126 lvb_type: 0 Jul 15 15:34:26 fir-md1-s1 kernel: LustreError: 20378:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 10 previous similar messages Jul 15 15:34:36 fir-md1-s1 kernel: LNet: Service thread pid 23077 completed after 209.64s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jul 15 15:38:28 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Mon Jul 15 15:38:28 2019