[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-957.27.2.el7_lustre.pl2.x86_64 (sthiell@oak-rbh01) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Thu Nov 7 15:26:16 PST 2019 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl2.x86_64 root=UUID=a6545b0a-ca64-4a56-96cf-d88ad9bb96eb ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000008efff] usable [ 0.000000] BIOS-e820: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x0000000000090000-0x000000000009ffff] usable [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000004f780fff] usable [ 0.000000] BIOS-e820: [mem 0x000000004f781000-0x0000000057789fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000005778a000-0x000000006cacefff] usable [ 0.000000] BIOS-e820: [mem 0x000000006cacf000-0x000000006efcefff] reserved [ 0.000000] BIOS-e820: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable [ 0.000000] BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed80000-0x00000000fed80fff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000107f37ffff] usable [ 0.000000] BIOS-e820: [mem 0x000000107f380000-0x000000107fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000001080000000-0x000000207ff7ffff] usable [ 0.000000] BIOS-e820: [mem 0x000000207ff80000-0x000000207fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000002080000000-0x000000307ff7ffff] usable [ 0.000000] BIOS-e820: [mem 0x000000307ff80000-0x000000307fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000003080000000-0x000000407ff7ffff] usable [ 0.000000] BIOS-e820: [mem 0x000000407ff80000-0x000000407fffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] e820: update [mem 0x3795e020-0x379ffc5f] usable ==> usable [ 0.000000] e820: update [mem 0x3792c020-0x3795dc5f] usable ==> usable [ 0.000000] e820: update [mem 0x378fa020-0x3792bc5f] usable ==> usable [ 0.000000] e820: update [mem 0x378f1020-0x378f905f] usable ==> usable [ 0.000000] e820: update [mem 0x378cb020-0x378f0c5f] usable ==> usable [ 0.000000] e820: update [mem 0x378b2020-0x378ca65f] usable ==> usable [ 0.000000] extended physical RAM map: [ 0.000000] reserve setup_data: [mem 0x0000000000000000-0x000000000008efff] usable [ 0.000000] reserve setup_data: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS [ 0.000000] reserve setup_data: [mem 0x0000000000090000-0x000000000009ffff] usable [ 0.000000] reserve setup_data: [mem 0x0000000000100000-0x00000000378b201f] usable [ 0.000000] reserve setup_data: [mem 0x00000000378b2020-0x00000000378ca65f] usable [ 0.000000] reserve setup_data: [mem 0x00000000378ca660-0x00000000378cb01f] usable [ 0.000000] reserve setup_data: [mem 0x00000000378cb020-0x00000000378f0c5f] usable [ 0.000000] reserve setup_data: [mem 0x00000000378f0c60-0x00000000378f101f] usable [ 0.000000] reserve setup_data: [mem 0x00000000378f1020-0x00000000378f905f] usable [ 0.000000] reserve setup_data: [mem 0x00000000378f9060-0x00000000378fa01f] usable [ 0.000000] reserve setup_data: [mem 0x00000000378fa020-0x000000003792bc5f] usable [ 0.000000] reserve setup_data: [mem 0x000000003792bc60-0x000000003792c01f] usable [ 0.000000] reserve setup_data: [mem 0x000000003792c020-0x000000003795dc5f] usable [ 0.000000] reserve setup_data: [mem 0x000000003795dc60-0x000000003795e01f] usable [ 0.000000] reserve setup_data: [mem 0x000000003795e020-0x00000000379ffc5f] usable [ 0.000000] reserve setup_data: [mem 0x00000000379ffc60-0x000000004f780fff] usable [ 0.000000] reserve setup_data: [mem 0x000000004f781000-0x0000000057789fff] reserved [ 0.000000] reserve setup_data: [mem 0x000000005778a000-0x000000006cacefff] usable [ 0.000000] reserve setup_data: [mem 0x000000006cacf000-0x000000006efcefff] reserved [ 0.000000] reserve setup_data: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS [ 0.000000] reserve setup_data: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data [ 0.000000] reserve setup_data: [mem 0x000000006ffff000-0x000000006fffffff] usable [ 0.000000] reserve setup_data: [mem 0x0000000070000000-0x000000008fffffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000fec10000-0x00000000fec10fff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000fed80000-0x00000000fed80fff] reserved [ 0.000000] reserve setup_data: [mem 0x0000000100000000-0x000000107f37ffff] usable [ 0.000000] reserve setup_data: [mem 0x000000107f380000-0x000000107fffffff] reserved [ 0.000000] reserve setup_data: [mem 0x0000001080000000-0x000000207ff7ffff] usable [ 0.000000] reserve setup_data: [mem 0x000000207ff80000-0x000000207fffffff] reserved [ 0.000000] reserve setup_data: [mem 0x0000002080000000-0x000000307ff7ffff] usable [ 0.000000] reserve setup_data: [mem 0x000000307ff80000-0x000000307fffffff] reserved [ 0.000000] reserve setup_data: [mem 0x0000003080000000-0x000000407ff7ffff] usable [ 0.000000] reserve setup_data: [mem 0x000000407ff80000-0x000000407fffffff] reserved [ 0.000000] efi: EFI v2.50 by Dell Inc. [ 0.000000] efi: ACPI=0x6fffe000 ACPI 2.0=0x6fffe014 SMBIOS=0x6eab5000 SMBIOS 3.0=0x6eab3000 [ 0.000000] efi: mem00: type=3, attr=0xf, range=[0x0000000000000000-0x0000000000001000) (0MB) [ 0.000000] efi: mem01: type=2, attr=0xf, range=[0x0000000000001000-0x0000000000002000) (0MB) [ 0.000000] efi: mem02: type=7, attr=0xf, range=[0x0000000000002000-0x0000000000010000) (0MB) [ 0.000000] efi: mem03: type=3, attr=0xf, range=[0x0000000000010000-0x0000000000014000) (0MB) [ 0.000000] efi: mem04: type=7, attr=0xf, range=[0x0000000000014000-0x0000000000063000) (0MB) [ 0.000000] efi: mem05: type=3, attr=0xf, range=[0x0000000000063000-0x000000000008f000) (0MB) [ 0.000000] efi: mem06: type=10, attr=0xf, range=[0x000000000008f000-0x0000000000090000) (0MB) [ 0.000000] efi: mem07: type=3, attr=0xf, range=[0x0000000000090000-0x00000000000a0000) (0MB) [ 0.000000] efi: mem08: type=4, attr=0xf, range=[0x0000000000100000-0x0000000000120000) (0MB) [ 0.000000] efi: mem09: type=7, attr=0xf, range=[0x0000000000120000-0x0000000000c00000) (10MB) [ 0.000000] efi: mem10: type=3, attr=0xf, range=[0x0000000000c00000-0x0000000001000000) (4MB) [ 0.000000] efi: mem11: type=2, attr=0xf, range=[0x0000000001000000-0x000000000267b000) (22MB) [ 0.000000] efi: mem12: type=7, attr=0xf, range=[0x000000000267b000-0x0000000004000000) (25MB) [ 0.000000] efi: mem13: type=4, attr=0xf, range=[0x0000000004000000-0x000000000403b000) (0MB) [ 0.000000] efi: mem14: type=7, attr=0xf, range=[0x000000000403b000-0x00000000378b2000) (824MB) [ 0.000000] efi: mem15: type=2, attr=0xf, range=[0x00000000378b2000-0x000000004ede4000) (373MB) [ 0.000000] efi: mem16: type=7, attr=0xf, range=[0x000000004ede4000-0x000000004ede8000) (0MB) [ 0.000000] efi: mem17: type=2, attr=0xf, range=[0x000000004ede8000-0x000000004edea000) (0MB) [ 0.000000] efi: mem18: type=1, attr=0xf, range=[0x000000004edea000-0x000000004ef07000) (1MB) [ 0.000000] efi: mem19: type=2, attr=0xf, range=[0x000000004ef07000-0x000000004f026000) (1MB) [ 0.000000] efi: mem20: type=1, attr=0xf, range=[0x000000004f026000-0x000000004f135000) (1MB) [ 0.000000] efi: mem21: type=3, attr=0xf, range=[0x000000004f135000-0x000000004f781000) (6MB) [ 0.000000] efi: mem22: type=0, attr=0xf, range=[0x000000004f781000-0x000000005778a000) (128MB) [ 0.000000] efi: mem23: type=3, attr=0xf, range=[0x000000005778a000-0x000000005796e000) (1MB) [ 0.000000] efi: mem24: type=4, attr=0xf, range=[0x000000005796e000-0x000000005b4cf000) (59MB) [ 0.000000] efi: mem25: type=3, attr=0xf, range=[0x000000005b4cf000-0x000000005b8cf000) (4MB) [ 0.000000] efi: mem26: type=7, attr=0xf, range=[0x000000005b8cf000-0x000000006531c000) (154MB) [ 0.000000] efi: mem27: type=4, attr=0xf, range=[0x000000006531c000-0x0000000065329000) (0MB) [ 0.000000] efi: mem28: type=7, attr=0xf, range=[0x0000000065329000-0x000000006532d000) (0MB) [ 0.000000] efi: mem29: type=4, attr=0xf, range=[0x000000006532d000-0x000000006595c000) (6MB) [ 0.000000] efi: mem30: type=7, attr=0xf, range=[0x000000006595c000-0x000000006595d000) (0MB) [ 0.000000] efi: mem31: type=4, attr=0xf, range=[0x000000006595d000-0x0000000065966000) (0MB) [ 0.000000] efi: mem32: type=7, attr=0xf, range=[0x0000000065966000-0x0000000065967000) (0MB) [ 0.000000] efi: mem33: type=4, attr=0xf, range=[0x0000000065967000-0x000000006597a000) (0MB) [ 0.000000] efi: mem34: type=7, attr=0xf, range=[0x000000006597a000-0x000000006597b000) (0MB) [ 0.000000] efi: mem35: type=4, attr=0xf, range=[0x000000006597b000-0x000000006597c000) (0MB) [ 0.000000] efi: mem36: type=7, attr=0xf, range=[0x000000006597c000-0x000000006597d000) (0MB) [ 0.000000] efi: mem37: type=4, attr=0xf, range=[0x000000006597d000-0x0000000065980000) (0MB) [ 0.000000] efi: mem38: type=7, attr=0xf, range=[0x0000000065980000-0x0000000065981000) (0MB) [ 0.000000] efi: mem39: type=4, attr=0xf, range=[0x0000000065981000-0x0000000065986000) (0MB) [ 0.000000] efi: mem40: type=7, attr=0xf, range=[0x0000000065986000-0x0000000065987000) (0MB) [ 0.000000] efi: mem41: type=4, attr=0xf, range=[0x0000000065987000-0x000000006598e000) (0MB) [ 0.000000] efi: mem42: type=7, attr=0xf, range=[0x000000006598e000-0x000000006598f000) (0MB) [ 0.000000] efi: mem43: type=4, attr=0xf, range=[0x000000006598f000-0x00000000659a1000) (0MB) [ 0.000000] efi: mem44: type=7, attr=0xf, range=[0x00000000659a1000-0x00000000659a2000) (0MB) [ 0.000000] efi: mem45: type=4, attr=0xf, range=[0x00000000659a2000-0x00000000659a6000) (0MB) [ 0.000000] efi: mem46: type=7, attr=0xf, range=[0x00000000659a6000-0x00000000659a7000) (0MB) [ 0.000000] efi: mem47: type=4, attr=0xf, range=[0x00000000659a7000-0x00000000659aa000) (0MB) [ 0.000000] efi: mem48: type=7, attr=0xf, range=[0x00000000659aa000-0x00000000659ab000) (0MB) [ 0.000000] efi: mem49: type=4, attr=0xf, range=[0x00000000659ab000-0x00000000659ac000) (0MB) [ 0.000000] efi: mem50: type=7, attr=0xf, range=[0x00000000659ac000-0x00000000659ad000) (0MB) [ 0.000000] efi: mem51: type=4, attr=0xf, range=[0x00000000659ad000-0x00000000659b0000) (0MB) [ 0.000000] efi: mem52: type=7, attr=0xf, range=[0x00000000659b0000-0x00000000659b1000) (0MB) [ 0.000000] efi: mem53: type=4, attr=0xf, range=[0x00000000659b1000-0x00000000659b5000) (0MB) [ 0.000000] efi: mem54: type=7, attr=0xf, range=[0x00000000659b5000-0x00000000659b6000) (0MB) [ 0.000000] efi: mem55: type=4, attr=0xf, range=[0x00000000659b6000-0x00000000659ba000) (0MB) [ 0.000000] efi: mem56: type=7, attr=0xf, range=[0x00000000659ba000-0x00000000659bb000) (0MB) [ 0.000000] efi: mem57: type=4, attr=0xf, range=[0x00000000659bb000-0x00000000659c2000) (0MB) [ 0.000000] efi: mem58: type=7, attr=0xf, range=[0x00000000659c2000-0x00000000659c3000) (0MB) [ 0.000000] efi: mem59: type=4, attr=0xf, range=[0x00000000659c3000-0x00000000659c4000) (0MB) [ 0.000000] efi: mem60: type=7, attr=0xf, range=[0x00000000659c4000-0x00000000659c5000) (0MB) [ 0.000000] efi: mem61: type=4, attr=0xf, range=[0x00000000659c5000-0x00000000659d5000) (0MB) [ 0.000000] efi: mem62: type=7, attr=0xf, range=[0x00000000659d5000-0x00000000659d6000) (0MB) [ 0.000000] efi: mem63: type=4, attr=0xf, range=[0x00000000659d6000-0x0000000065a58000) (0MB) [ 0.000000] efi: mem64: type=7, attr=0xf, range=[0x0000000065a58000-0x0000000065a59000) (0MB) [ 0.000000] efi: mem65: type=4, attr=0xf, range=[0x0000000065a59000-0x0000000065ceb000) (2MB) [ 0.000000] efi: mem66: type=7, attr=0xf, range=[0x0000000065ceb000-0x0000000065cec000) (0MB) [ 0.000000] efi: mem67: type=4, attr=0xf, range=[0x0000000065cec000-0x0000000065d1c000) (0MB) [ 0.000000] efi: mem68: type=7, attr=0xf, range=[0x0000000065d1c000-0x0000000065d1d000) (0MB) [ 0.000000] efi: mem69: type=4, attr=0xf, range=[0x0000000065d1d000-0x0000000065d30000) (0MB) [ 0.000000] efi: mem70: type=7, attr=0xf, range=[0x0000000065d30000-0x0000000065d31000) (0MB) [ 0.000000] efi: mem71: type=4, attr=0xf, range=[0x0000000065d31000-0x0000000065d73000) (0MB) [ 0.000000] efi: mem72: type=7, attr=0xf, range=[0x0000000065d73000-0x0000000065d74000) (0MB) [ 0.000000] efi: mem73: type=4, attr=0xf, range=[0x0000000065d74000-0x0000000065da8000) (0MB) [ 0.000000] efi: mem74: type=7, attr=0xf, range=[0x0000000065da8000-0x0000000065da9000) (0MB) [ 0.000000] efi: mem75: type=4, attr=0xf, range=[0x0000000065da9000-0x0000000065dc5000) (0MB) [ 0.000000] efi: mem76: type=7, attr=0xf, range=[0x0000000065dc5000-0x0000000065dc6000) (0MB) [ 0.000000] efi: mem77: type=4, attr=0xf, range=[0x0000000065dc6000-0x0000000065dd4000) (0MB) [ 0.000000] efi: mem78: type=7, attr=0xf, range=[0x0000000065dd4000-0x0000000065dd5000) (0MB) [ 0.000000] efi: mem79: type=4, attr=0xf, range=[0x0000000065dd5000-0x0000000065df4000) (0MB) [ 0.000000] efi: mem80: type=7, attr=0xf, range=[0x0000000065df4000-0x0000000065df5000) (0MB) [ 0.000000] efi: mem81: type=4, attr=0xf, range=[0x0000000065df5000-0x0000000065e01000) (0MB) [ 0.000000] efi: mem82: type=7, attr=0xf, range=[0x0000000065e01000-0x0000000065e02000) (0MB) [ 0.000000] efi: mem83: type=4, attr=0xf, range=[0x0000000065e02000-0x0000000065e08000) (0MB) [ 0.000000] efi: mem84: type=7, attr=0xf, range=[0x0000000065e08000-0x0000000065e09000) (0MB) [ 0.000000] efi: mem85: type=4, attr=0xf, range=[0x0000000065e09000-0x0000000065e64000) (0MB) [ 0.000000] efi: mem86: type=7, attr=0xf, range=[0x0000000065e64000-0x0000000065e66000) (0MB) [ 0.000000] efi: mem87: type=4, attr=0xf, range=[0x0000000065e66000-0x0000000065e84000) (0MB) [ 0.000000] efi: mem88: type=7, attr=0xf, range=[0x0000000065e84000-0x0000000065e85000) (0MB) [ 0.000000] efi: mem89: type=4, attr=0xf, range=[0x0000000065e85000-0x0000000065e95000) (0MB) [ 0.000000] efi: mem90: type=7, attr=0xf, range=[0x0000000065e95000-0x0000000065e96000) (0MB) [ 0.000000] efi: mem91: type=4, attr=0xf, range=[0x0000000065e96000-0x0000000065eb1000) (0MB) [ 0.000000] efi: mem92: type=7, attr=0xf, range=[0x0000000065eb1000-0x0000000065eb2000) (0MB) [ 0.000000] efi: mem93: type=4, attr=0xf, range=[0x0000000065eb2000-0x0000000065ec1000) (0MB) [ 0.000000] efi: mem94: type=7, attr=0xf, range=[0x0000000065ec1000-0x0000000065ec2000) (0MB) [ 0.000000] efi: mem95: type=4, attr=0xf, range=[0x0000000065ec2000-0x0000000065eca000) (0MB) [ 0.000000] efi: mem96: type=7, attr=0xf, range=[0x0000000065eca000-0x0000000065ecb000) (0MB) [ 0.000000] efi: mem97: type=4, attr=0xf, range=[0x0000000065ecb000-0x000000006b8cf000) (90MB) [ 0.000000] efi: mem98: type=7, attr=0xf, range=[0x000000006b8cf000-0x000000006b8d0000) (0MB) [ 0.000000] efi: mem99: type=3, attr=0xf, range=[0x000000006b8d0000-0x000000006cacf000) (17MB) [ 0.000000] efi: mem100: type=6, attr=0x800000000000000f, range=[0x000000006cacf000-0x000000006cbcf000) (1MB) [ 0.000000] efi: mem101: type=5, attr=0x800000000000000f, range=[0x000000006cbcf000-0x000000006cdcf000) (2MB) [ 0.000000] efi: mem102: type=0, attr=0xf, range=[0x000000006cdcf000-0x000000006efcf000) (34MB) [ 0.000000] efi: mem103: type=10, attr=0xf, range=[0x000000006efcf000-0x000000006fdff000) (14MB) [ 0.000000] efi: mem104: type=9, attr=0xf, range=[0x000000006fdff000-0x000000006ffff000) (2MB) [ 0.000000] efi: mem105: type=4, attr=0xf, range=[0x000000006ffff000-0x0000000070000000) (0MB) [ 0.000000] efi: mem106: type=7, attr=0xf, range=[0x0000000100000000-0x000000107f380000) (63475MB) [ 0.000000] efi: mem107: type=7, attr=0xf, range=[0x0000001080000000-0x000000207ff80000) (65535MB) [ 0.000000] efi: mem108: type=7, attr=0xf, range=[0x0000002080000000-0x000000307ff80000) (65535MB) [ 0.000000] efi: mem109: type=7, attr=0xf, range=[0x0000003080000000-0x000000407ff80000) (65535MB) [ 0.000000] efi: mem110: type=0, attr=0x9, range=[0x0000000070000000-0x0000000080000000) (256MB) [ 0.000000] efi: mem111: type=11, attr=0x800000000000000f, range=[0x0000000080000000-0x0000000090000000) (256MB) [ 0.000000] efi: mem112: type=11, attr=0x800000000000000f, range=[0x00000000fec10000-0x00000000fec11000) (0MB) [ 0.000000] efi: mem113: type=11, attr=0x800000000000000f, range=[0x00000000fed80000-0x00000000fed81000) (0MB) [ 0.000000] efi: mem114: type=0, attr=0x0, range=[0x000000107f380000-0x0000001080000000) (12MB) [ 0.000000] efi: mem115: type=0, attr=0x0, range=[0x000000207ff80000-0x0000002080000000) (0MB) [ 0.000000] efi: mem116: type=0, attr=0x0, range=[0x000000307ff80000-0x0000003080000000) (0MB) [ 0.000000] efi: mem117: type=0, attr=0x0, range=[0x000000407ff80000-0x0000004080000000) (0MB) [ 0.000000] SMBIOS 3.2.0 present. [ 0.000000] DMI: Dell Inc. PowerEdge R6415/07YXFK, BIOS 1.10.6 08/15/2019 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x407ff80 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-FFFFF uncachable [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000FF000000 mask FFFFFF000000 write-protect [ 0.000000] 1 base 000000000000 mask FFFF80000000 write-back [ 0.000000] 2 base 000070000000 mask FFFFF0000000 uncachable [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] TOM2: 0000004080000000 aka 264192M [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0x70000 max_arch_pfn = 0x400000000 [ 0.000000] Base memory trampoline at [ffff8b6b80099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x3944e53000, 0x3944e53fff] PGTABLE [ 0.000000] BRK [0x3944e54000, 0x3944e54fff] PGTABLE [ 0.000000] BRK [0x3944e55000, 0x3944e55fff] PGTABLE [ 0.000000] BRK [0x3944e56000, 0x3944e56fff] PGTABLE [ 0.000000] BRK [0x3944e57000, 0x3944e57fff] PGTABLE [ 0.000000] BRK [0x3944e58000, 0x3944e58fff] PGTABLE [ 0.000000] BRK [0x3944e59000, 0x3944e59fff] PGTABLE [ 0.000000] BRK [0x3944e5a000, 0x3944e5afff] PGTABLE [ 0.000000] BRK [0x3944e5b000, 0x3944e5bfff] PGTABLE [ 0.000000] BRK [0x3944e5c000, 0x3944e5cfff] PGTABLE [ 0.000000] BRK [0x3944e5d000, 0x3944e5dfff] PGTABLE [ 0.000000] BRK [0x3944e5e000, 0x3944e5efff] PGTABLE [ 0.000000] RAMDISK: [mem 0x37a00000-0x38d22fff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 000000006fffe014 00024 (v02 DELL ) [ 0.000000] ACPI: XSDT 000000006fffd0e8 000AC (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: FACP 000000006fff0000 00114 (v06 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: DSDT 000000006ffdc000 1038C (v02 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: FACS 000000006fdd3000 00040 [ 0.000000] ACPI: SSDT 000000006fffc000 000D2 (v02 DELL PE_SC3 00000002 MSFT 04000000) [ 0.000000] ACPI: BERT 000000006fffb000 00030 (v01 DELL BERT 00000001 DELL 00000001) [ 0.000000] ACPI: HEST 000000006fffa000 006DC (v01 DELL HEST 00000001 DELL 00000001) [ 0.000000] ACPI: SSDT 000000006fff9000 00294 (v01 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: SRAT 000000006fff8000 00420 (v03 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: MSCT 000000006fff7000 0004E (v01 DELL PE_SC3 00000000 AMD 00000001) [ 0.000000] ACPI: SLIT 000000006fff6000 0003C (v01 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: CRAT 000000006fff3000 02DC0 (v01 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: EINJ 000000006fff2000 00150 (v01 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: SLIC 000000006fff1000 00024 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: HPET 000000006ffef000 00038 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: APIC 000000006ffee000 004B2 (v03 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: MCFG 000000006ffed000 0003C (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: SSDT 000000006ffdb000 00629 (v02 DELL xhc_port 00000001 INTL 20170119) [ 0.000000] ACPI: IVRS 000000006ffda000 00210 (v02 DELL PE_SC3 00000001 AMD 00000000) [ 0.000000] ACPI: SSDT 000000006ffd8000 01658 (v01 AMD CPMCMN 00000001 INTL 20170119) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0a -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0b -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0c -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0d -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x10 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x11 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x12 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x13 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x14 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x15 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x18 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x19 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x1a -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x1b -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x1c -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x1d -> Node 1 [ 0.000000] SRAT: PXM 2 -> APIC 0x20 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x21 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x22 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x23 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x24 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x25 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x28 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x29 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x2a -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x2b -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x2c -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x2d -> Node 2 [ 0.000000] SRAT: PXM 3 -> APIC 0x30 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x31 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x32 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x33 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x34 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x35 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x38 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x39 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x3a -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x3b -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x3c -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x3d -> Node 3 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0x7fffffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff] [ 0.000000] SRAT: Node 2 PXM 2 [mem 0x2080000000-0x307fffffff] [ 0.000000] SRAT: Node 3 PXM 3 [mem 0x3080000000-0x407fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=4 [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x7fffffff] -> [mem 0x00000000-0x7fffffff] [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [mem 0x00000000-0x107fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x107f359000-0x107f37ffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x207ff59000-0x207ff7ffff] [ 0.000000] NODE_DATA(2) allocated [mem 0x307ff59000-0x307ff7ffff] [ 0.000000] NODE_DATA(3) allocated [mem 0x407ff58000-0x407ff7efff] [ 0.000000] Reserving 176MB of memory at 704MB for crashkernel (System RAM: 261692MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x407ff7ffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0008efff] [ 0.000000] node 0: [mem 0x00090000-0x0009ffff] [ 0.000000] node 0: [mem 0x00100000-0x4f780fff] [ 0.000000] node 0: [mem 0x5778a000-0x6cacefff] [ 0.000000] node 0: [mem 0x6ffff000-0x6fffffff] [ 0.000000] node 0: [mem 0x100000000-0x107f37ffff] [ 0.000000] node 1: [mem 0x1080000000-0x207ff7ffff] [ 0.000000] node 2: [mem 0x2080000000-0x307ff7ffff] [ 0.000000] node 3: [mem 0x3080000000-0x407ff7ffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x107f37ffff] [ 0.000000] On node 0 totalpages: 16661989 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 1126 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 6380 pages used for memmap [ 0.000000] DMA32 zone: 408263 pages, LIFO batch:31 [ 0.000000] Normal zone: 253902 pages used for memmap [ 0.000000] Normal zone: 16249728 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x1080000000-0x207ff7ffff] [ 0.000000] On node 1 totalpages: 16777088 [ 0.000000] Normal zone: 262142 pages used for memmap [ 0.000000] Normal zone: 16777088 pages, LIFO batch:31 [ 0.000000] Initmem setup node 2 [mem 0x2080000000-0x307ff7ffff] [ 0.000000] On node 2 totalpages: 16777088 [ 0.000000] Normal zone: 262142 pages used for memmap [ 0.000000] Normal zone: 16777088 pages, LIFO batch:31 [ 0.000000] Initmem setup node 3 [mem 0x3080000000-0x407ff7ffff] [ 0.000000] On node 3 totalpages: 16777088 [ 0.000000] Normal zone: 262142 pages used for memmap [ 0.000000] Normal zone: 16777088 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x20] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x30] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x18] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x28] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x38] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x12] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x22] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x1a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x3a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x14] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x24] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x34] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x0c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x1c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x2c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x3c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x11] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x21] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x31] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x19] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x29] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x39] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0x13] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0x23] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0x33] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0x0b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0x1b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0x2b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0x3b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0x15] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x25] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x35] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x0d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x1d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x2d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x3d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x40] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x41] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x42] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x43] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x44] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x45] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x46] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x47] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x48] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x49] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x50] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x51] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x52] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x53] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x54] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x55] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x56] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x57] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x58] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x59] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x60] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x61] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x62] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x63] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x64] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x65] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x66] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x67] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x68] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x69] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x70] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x71] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x72] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x73] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x74] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x75] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x76] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x77] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x78] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x79] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x80] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x81] address[0xfd880000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 129, version 33, address 0xfd880000, GSI 24-55 [ 0.000000] ACPI: IOAPIC (id[0x82] address[0xe0900000] gsi_base[56]) [ 0.000000] IOAPIC[2]: apic_id 130, version 33, address 0xe0900000, GSI 56-87 [ 0.000000] ACPI: IOAPIC (id[0x83] address[0xc5900000] gsi_base[88]) [ 0.000000] IOAPIC[3]: apic_id 131, version 33, address 0xc5900000, GSI 88-119 [ 0.000000] ACPI: IOAPIC (id[0x84] address[0xaa900000] gsi_base[120]) [ 0.000000] IOAPIC[4]: apic_id 132, version 33, address 0xaa900000, GSI 120-151 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x10228201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 128 CPUs, 80 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0008f000-0x0008ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x378b2000-0x378b2fff] [ 0.000000] PM: Registered nosave memory: [mem 0x378ca000-0x378cafff] [ 0.000000] PM: Registered nosave memory: [mem 0x378cb000-0x378cbfff] [ 0.000000] PM: Registered nosave memory: [mem 0x378f0000-0x378f0fff] [ 0.000000] PM: Registered nosave memory: [mem 0x378f1000-0x378f1fff] [ 0.000000] PM: Registered nosave memory: [mem 0x378f9000-0x378f9fff] [ 0.000000] PM: Registered nosave memory: [mem 0x378fa000-0x378fafff] [ 0.000000] PM: Registered nosave memory: [mem 0x3792b000-0x3792bfff] [ 0.000000] PM: Registered nosave memory: [mem 0x3792c000-0x3792cfff] [ 0.000000] PM: Registered nosave memory: [mem 0x3795d000-0x3795dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x3795e000-0x3795efff] [ 0.000000] PM: Registered nosave memory: [mem 0x379ff000-0x379fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x4f781000-0x57789fff] [ 0.000000] PM: Registered nosave memory: [mem 0x6cacf000-0x6efcefff] [ 0.000000] PM: Registered nosave memory: [mem 0x6efcf000-0x6fdfefff] [ 0.000000] PM: Registered nosave memory: [mem 0x6fdff000-0x6fffefff] [ 0.000000] PM: Registered nosave memory: [mem 0x70000000-0x8fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x90000000-0xfec0ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec10000-0xfec10fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec11000-0xfed7ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed80000-0xfed80fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed81000-0xffffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x107f380000-0x107fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x207ff80000-0x207fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x307ff80000-0x307fffffff] [ 0.000000] e820: [mem 0x90000000-0xfec0ffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:4 [ 0.000000] PERCPU: Embedded 38 pages/cpu @ffff8b7bbee00000 s118784 r8192 d28672 u262144 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 000 004 008 012 016 020 024 028 [ 0.000000] pcpu-alloc: [0] 032 036 040 044 048 052 056 060 [ 0.000000] pcpu-alloc: [0] 064 068 072 076 080 084 088 092 [ 0.000000] pcpu-alloc: [0] 096 100 104 108 112 116 120 124 [ 0.000000] pcpu-alloc: [1] 001 005 009 013 017 021 025 029 [ 0.000000] pcpu-alloc: [1] 033 037 041 045 049 053 057 061 [ 0.000000] pcpu-alloc: [1] 065 069 073 077 081 085 089 093 [ 0.000000] pcpu-alloc: [1] 097 101 105 109 113 117 121 125 [ 0.000000] pcpu-alloc: [2] 002 006 010 014 018 022 026 030 [ 0.000000] pcpu-alloc: [2] 034 038 042 046 050 054 058 062 [ 0.000000] pcpu-alloc: [2] 066 070 074 078 082 086 090 094 [ 0.000000] pcpu-alloc: [2] 098 102 106 110 114 118 122 126 [ 0.000000] pcpu-alloc: [3] 003 007 011 015 019 023 027 031 [ 0.000000] pcpu-alloc: [3] 035 039 043 047 051 055 059 063 [ 0.000000] pcpu-alloc: [3] 067 071 075 079 083 087 091 095 [ 0.000000] pcpu-alloc: [3] 099 103 107 111 115 119 123 127 [ 0.000000] Built 4 zonelists in Zone order, mobility grouping on. Total pages: 65945355 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl2.x86_64 root=UUID=a6545b0a-ca64-4a56-96cf-d88ad9bb96eb ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 9570480k/270532096k available (7676k kernel code, 2559084k absent, 4697484k reserved, 6045k data, 1876k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=4 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=128. [ 0.000000] NR_IRQS:327936 nr_irqs:3624 0 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 1072693248 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 1996.201 MHz processor [ 0.000055] Calibrating delay loop (skipped), value calculated using timer frequency.. 3992.40 BogoMIPS (lpj=1996201) [ 0.010706] pid_max: default: 131072 minimum: 1024 [ 0.016323] Security Framework initialized [ 0.020446] SELinux: Initializing. [ 0.024004] SELinux: Starting in permissive mode [ 0.024005] Yama: becoming mindful. [ 0.044068] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) [ 0.099931] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.127665] Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.135064] Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.144146] Initializing cgroup subsys memory [ 0.148545] Initializing cgroup subsys devices [ 0.153006] Initializing cgroup subsys freezer [ 0.157462] Initializing cgroup subsys net_cls [ 0.161916] Initializing cgroup subsys blkio [ 0.166197] Initializing cgroup subsys perf_event [ 0.170920] Initializing cgroup subsys hugetlb [ 0.175375] Initializing cgroup subsys pids [ 0.179570] Initializing cgroup subsys net_prio [ 0.184183] tseg: 0070000000 [ 0.189809] LVT offset 2 assigned for vector 0xf4 [ 0.194542] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 512 [ 0.200562] Last level dTLB entries: 4KB 1536, 2MB 1536, 4MB 768 [ 0.206577] tlb_flushall_shift: 6 [ 0.209926] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.219498] FEATURE SPEC_CTRL Not Present [ 0.223520] FEATURE IBPB_SUPPORT Present [ 0.227456] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.233892] Spectre V2 : Mitigation: Full retpoline [ 0.239198] Freeing SMP alternatives: 28k freed [ 0.245617] ACPI: Core revision 20130517 [ 0.254304] ACPI: All ACPI Tables successfully acquired [ 0.265930] ftrace: allocating 29216 entries in 115 pages [ 0.606259] Switched APIC routing to physical flat. [ 0.613183] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.629192] smpboot: CPU0: AMD EPYC 7401P 24-Core Processor (fam: 17, model: 01, stepping: 02) [ 0.714406] random: fast init done [ 0.741407] APIC calibration not consistent with PM-Timer: 101ms instead of 100ms [ 0.748888] APIC delta adjusted to PM-Timer: 623827 (636297) [ 0.754579] Performance Events: Fam17h core perfctr, AMD PMU driver. [ 0.761016] ... version: 0 [ 0.765027] ... bit width: 48 [ 0.769125] ... generic registers: 6 [ 0.773137] ... value mask: 0000ffffffffffff [ 0.778452] ... max period: 00007fffffffffff [ 0.783763] ... fixed-purpose events: 0 [ 0.787776] ... event mask: 000000000000003f [ 0.796108] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.804188] smpboot: Booting Node 1, Processors #1 OK [ 0.817399] smpboot: Booting Node 2, Processors #2 OK [ 0.830611] smpboot: Booting Node 3, Processors #3 OK [ 0.843820] smpboot: Booting Node 0, Processors #4 OK [ 0.857002] smpboot: Booting Node 1, Processors #5 OK [ 0.870180] smpboot: Booting Node 2, Processors #6 OK [ 0.883354] smpboot: Booting Node 3, Processors #7 OK [ 0.896529] smpboot: Booting Node 0, Processors #8 OK [ 0.909922] smpboot: Booting Node 1, Processors #9 OK [ 0.923118] smpboot: Booting Node 2, Processors #10 OK [ 0.936396] smpboot: Booting Node 3, Processors #11 OK [ 0.949668] smpboot: Booting Node 0, Processors #12 OK [ 0.962938] smpboot: Booting Node 1, Processors #13 OK [ 0.976211] smpboot: Booting Node 2, Processors #14 OK [ 0.989480] smpboot: Booting Node 3, Processors #15 OK [ 1.002754] smpboot: Booting Node 0, Processors #16 OK [ 1.016132] smpboot: Booting Node 1, Processors #17 OK [ 1.029409] smpboot: Booting Node 2, Processors #18 OK [ 1.042690] smpboot: Booting Node 3, Processors #19 OK [ 1.055958] smpboot: Booting Node 0, Processors #20 OK [ 1.069225] smpboot: Booting Node 1, Processors #21 OK [ 1.082493] smpboot: Booting Node 2, Processors #22 OK [ 1.095776] smpboot: Booting Node 3, Processors #23 OK [ 1.109044] smpboot: Booting Node 0, Processors #24 OK [ 1.122782] smpboot: Booting Node 1, Processors #25 OK [ 1.136025] smpboot: Booting Node 2, Processors #26 OK [ 1.149265] smpboot: Booting Node 3, Processors #27 OK [ 1.162489] smpboot: Booting Node 0, Processors #28 OK [ 1.175718] smpboot: Booting Node 1, Processors #29 OK [ 1.188953] smpboot: Booting Node 2, Processors #30 OK [ 1.202185] smpboot: Booting Node 3, Processors #31 OK [ 1.215411] smpboot: Booting Node 0, Processors #32 OK [ 1.228742] smpboot: Booting Node 1, Processors #33 OK [ 1.241985] smpboot: Booting Node 2, Processors #34 OK [ 1.255329] smpboot: Booting Node 3, Processors #35 OK [ 1.268555] smpboot: Booting Node 0, Processors #36 OK [ 1.281792] smpboot: Booting Node 1, Processors #37 OK [ 1.295035] smpboot: Booting Node 2, Processors #38 OK [ 1.308380] smpboot: Booting Node 3, Processors #39 OK [ 1.321605] smpboot: Booting Node 0, Processors #40 OK [ 1.334937] smpboot: Booting Node 1, Processors #41 OK [ 1.348282] smpboot: Booting Node 2, Processors #42 OK [ 1.361518] smpboot: Booting Node 3, Processors #43 OK [ 1.374745] smpboot: Booting Node 0, Processors #44 OK [ 1.387971] smpboot: Booting Node 1, Processors #45 OK [ 1.401203] smpboot: Booting Node 2, Processors #46 OK [ 1.414444] smpboot: Booting Node 3, Processors #47 [ 1.427252] Brought up 48 CPUs [ 1.430510] smpboot: Max logical packages: 3 [ 1.434787] smpboot: Total of 48 processors activated (191635.29 BogoMIPS) [ 1.723480] node 0 initialised, 15462980 pages in 274ms [ 1.732011] node 2 initialised, 15989367 pages in 278ms [ 1.732136] node 1 initialised, 15989367 pages in 278ms [ 1.736493] node 3 initialised, 15984544 pages in 282ms [ 1.748239] devtmpfs: initialized [ 1.774074] EVM: security.selinux [ 1.777393] EVM: security.ima [ 1.780363] EVM: security.capability [ 1.784042] PM: Registering ACPI NVS region [mem 0x0008f000-0x0008ffff] (4096 bytes) [ 1.791783] PM: Registering ACPI NVS region [mem 0x6efcf000-0x6fdfefff] (14876672 bytes) [ 1.801443] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.808318] pinctrl core: initialized pinctrl subsystem [ 1.813650] RTC time: 18:02:55, date: 03/18/20 [ 1.818257] NET: Registered protocol family 16 [ 1.823065] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.830634] ACPI: bus type PCI registered [ 1.834646] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.841231] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [ 1.850534] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 [ 1.857326] PCI: Using configuration type 1 for base access [ 1.862911] PCI: Dell System detected, enabling pci=bfsort. [ 1.878013] ACPI: Added _OSI(Module Device) [ 1.882206] ACPI: Added _OSI(Processor Device) [ 1.886657] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.891360] ACPI: Added _OSI(Processor Aggregator Device) [ 1.896761] ACPI: Added _OSI(Linux-Dell-Video) [ 1.902027] ACPI: EC: Look up EC in DSDT [ 1.903007] ACPI: Executed 2 blocks of module-level executable AML code [ 1.915056] ACPI: Interpreter enabled [ 1.918731] ACPI: (supports S0 S5) [ 1.922137] ACPI: Using IOAPIC for interrupt routing [ 1.927314] HEST: Table parsing has been initialized. [ 1.932368] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.941515] ACPI: Enabled 1 GPEs in block 00 to 1F [ 1.953189] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 5 7 10 11 14 15) *0 [ 1.960100] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 5 7 10 11 14 15) *0 [ 1.967006] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 5 7 10 11 14 15) *0 [ 1.973911] ACPI: PCI Interrupt Link [LNKD] (IRQs 4 5 7 10 11 14 15) *0 [ 1.980820] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 5 7 10 11 14 15) *0 [ 1.987730] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 7 10 11 14 15) *0 [ 1.994636] ACPI: PCI Interrupt Link [LNKG] (IRQs 4 5 7 10 11 14 15) *0 [ 2.001543] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 7 10 11 14 15) *0 [ 2.008594] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00-3f]) [ 2.014784] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.023002] acpi PNP0A08:00: PCIe AER handled by firmware [ 2.028445] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug] [ 2.035392] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.043042] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.051500] PCI host bridge to bus 0000:00 [ 2.055601] pci_bus 0000:00: root bus resource [io 0x0000-0x03af window] [ 2.062386] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] [ 2.069171] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff window] [ 2.076651] pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window] [ 2.084130] pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff window] [ 2.091610] pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff window] [ 2.099091] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window] [ 2.106569] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window] [ 2.114049] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window] [ 2.121529] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window] [ 2.129009] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window] [ 2.136490] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window] [ 2.143968] pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff window] [ 2.151447] pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff window] [ 2.158927] pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff window] [ 2.166407] pci_bus 0000:00: root bus resource [io 0x0d00-0x3fff window] [ 2.173192] pci_bus 0000:00: root bus resource [mem 0xe1000000-0xfebfffff window] [ 2.180672] pci_bus 0000:00: root bus resource [mem 0x10000000000-0x2bf3fffffff window] [ 2.188672] pci_bus 0000:00: root bus resource [bus 00-3f] [ 2.194166] pci 0000:00:00.0: [1022:1450] type 00 class 0x060000 [ 2.194249] pci 0000:00:00.2: [1022:1451] type 00 class 0x080600 [ 2.194337] pci 0000:00:01.0: [1022:1452] type 00 class 0x060000 [ 2.194413] pci 0000:00:02.0: [1022:1452] type 00 class 0x060000 [ 2.194490] pci 0000:00:03.0: [1022:1452] type 00 class 0x060000 [ 2.194551] pci 0000:00:03.1: [1022:1453] type 01 class 0x060400 [ 2.195336] pci 0000:00:03.1: PME# supported from D0 D3hot D3cold [ 2.195434] pci 0000:00:04.0: [1022:1452] type 00 class 0x060000 [ 2.195518] pci 0000:00:07.0: [1022:1452] type 00 class 0x060000 [ 2.195579] pci 0000:00:07.1: [1022:1454] type 01 class 0x060400 [ 2.196330] pci 0000:00:07.1: PME# supported from D0 D3hot D3cold [ 2.196409] pci 0000:00:08.0: [1022:1452] type 00 class 0x060000 [ 2.196474] pci 0000:00:08.1: [1022:1454] type 01 class 0x060400 [ 2.197314] pci 0000:00:08.1: PME# supported from D0 D3hot D3cold [ 2.197429] pci 0000:00:14.0: [1022:790b] type 00 class 0x0c0500 [ 2.197630] pci 0000:00:14.3: [1022:790e] type 00 class 0x060100 [ 2.197835] pci 0000:00:18.0: [1022:1460] type 00 class 0x060000 [ 2.197887] pci 0000:00:18.1: [1022:1461] type 00 class 0x060000 [ 2.197938] pci 0000:00:18.2: [1022:1462] type 00 class 0x060000 [ 2.197988] pci 0000:00:18.3: [1022:1463] type 00 class 0x060000 [ 2.198038] pci 0000:00:18.4: [1022:1464] type 00 class 0x060000 [ 2.198089] pci 0000:00:18.5: [1022:1465] type 00 class 0x060000 [ 2.198140] pci 0000:00:18.6: [1022:1466] type 00 class 0x060000 [ 2.198191] pci 0000:00:18.7: [1022:1467] type 00 class 0x060000 [ 2.198242] pci 0000:00:19.0: [1022:1460] type 00 class 0x060000 [ 2.198296] pci 0000:00:19.1: [1022:1461] type 00 class 0x060000 [ 2.198352] pci 0000:00:19.2: [1022:1462] type 00 class 0x060000 [ 2.198406] pci 0000:00:19.3: [1022:1463] type 00 class 0x060000 [ 2.198462] pci 0000:00:19.4: [1022:1464] type 00 class 0x060000 [ 2.198516] pci 0000:00:19.5: [1022:1465] type 00 class 0x060000 [ 2.198570] pci 0000:00:19.6: [1022:1466] type 00 class 0x060000 [ 2.198627] pci 0000:00:19.7: [1022:1467] type 00 class 0x060000 [ 2.198680] pci 0000:00:1a.0: [1022:1460] type 00 class 0x060000 [ 2.198734] pci 0000:00:1a.1: [1022:1461] type 00 class 0x060000 [ 2.198786] pci 0000:00:1a.2: [1022:1462] type 00 class 0x060000 [ 2.198841] pci 0000:00:1a.3: [1022:1463] type 00 class 0x060000 [ 2.198894] pci 0000:00:1a.4: [1022:1464] type 00 class 0x060000 [ 2.198947] pci 0000:00:1a.5: [1022:1465] type 00 class 0x060000 [ 2.199002] pci 0000:00:1a.6: [1022:1466] type 00 class 0x060000 [ 2.199056] pci 0000:00:1a.7: [1022:1467] type 00 class 0x060000 [ 2.199109] pci 0000:00:1b.0: [1022:1460] type 00 class 0x060000 [ 2.199163] pci 0000:00:1b.1: [1022:1461] type 00 class 0x060000 [ 2.199216] pci 0000:00:1b.2: [1022:1462] type 00 class 0x060000 [ 2.199271] pci 0000:00:1b.3: [1022:1463] type 00 class 0x060000 [ 2.199324] pci 0000:00:1b.4: [1022:1464] type 00 class 0x060000 [ 2.199378] pci 0000:00:1b.5: [1022:1465] type 00 class 0x060000 [ 2.199432] pci 0000:00:1b.6: [1022:1466] type 00 class 0x060000 [ 2.199488] pci 0000:00:1b.7: [1022:1467] type 00 class 0x060000 [ 2.200359] pci 0000:01:00.0: [15b3:101b] type 00 class 0x020700 [ 2.200506] pci 0000:01:00.0: reg 0x10: [mem 0xe2000000-0xe3ffffff 64bit pref] [ 2.200741] pci 0000:01:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 2.201150] pci 0000:01:00.0: PME# supported from D3cold [ 2.201427] pci 0000:00:03.1: PCI bridge to [bus 01] [ 2.206402] pci 0000:00:03.1: bridge window [mem 0xe2000000-0xe3ffffff 64bit pref] [ 2.206478] pci 0000:02:00.0: [1022:145a] type 00 class 0x130000 [ 2.206575] pci 0000:02:00.2: [1022:1456] type 00 class 0x108000 [ 2.206593] pci 0000:02:00.2: reg 0x18: [mem 0xf7300000-0xf73fffff] [ 2.206605] pci 0000:02:00.2: reg 0x24: [mem 0xf7400000-0xf7401fff] [ 2.206684] pci 0000:02:00.3: [1022:145f] type 00 class 0x0c0330 [ 2.206696] pci 0000:02:00.3: reg 0x10: [mem 0xf7200000-0xf72fffff 64bit] [ 2.206744] pci 0000:02:00.3: PME# supported from D0 D3hot D3cold [ 2.206803] pci 0000:00:07.1: PCI bridge to [bus 02] [ 2.211773] pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] [ 2.212341] pci 0000:03:00.0: [1022:1455] type 00 class 0x130000 [ 2.212451] pci 0000:03:00.1: [1022:1468] type 00 class 0x108000 [ 2.212469] pci 0000:03:00.1: reg 0x18: [mem 0xf7000000-0xf70fffff] [ 2.212482] pci 0000:03:00.1: reg 0x24: [mem 0xf7100000-0xf7101fff] [ 2.212574] pci 0000:00:08.1: PCI bridge to [bus 03] [ 2.217545] pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] [ 2.217561] pci_bus 0000:00: on NUMA node 0 [ 2.217936] ACPI: PCI Root Bridge [PC01] (domain 0000 [bus 40-7f]) [ 2.224122] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.232331] acpi PNP0A08:01: PCIe AER handled by firmware [ 2.237774] acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug] [ 2.244722] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.252374] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.260788] PCI host bridge to bus 0000:40 [ 2.264891] pci_bus 0000:40: root bus resource [io 0x4000-0x7fff window] [ 2.271676] pci_bus 0000:40: root bus resource [mem 0xc6000000-0xe0ffffff window] [ 2.279153] pci_bus 0000:40: root bus resource [mem 0x2bf40000000-0x47e7fffffff window] [ 2.287154] pci_bus 0000:40: root bus resource [bus 40-7f] [ 2.292643] pci 0000:40:00.0: [1022:1450] type 00 class 0x060000 [ 2.292714] pci 0000:40:00.2: [1022:1451] type 00 class 0x080600 [ 2.292806] pci 0000:40:01.0: [1022:1452] type 00 class 0x060000 [ 2.292880] pci 0000:40:02.0: [1022:1452] type 00 class 0x060000 [ 2.292956] pci 0000:40:03.0: [1022:1452] type 00 class 0x060000 [ 2.293030] pci 0000:40:04.0: [1022:1452] type 00 class 0x060000 [ 2.293110] pci 0000:40:07.0: [1022:1452] type 00 class 0x060000 [ 2.293171] pci 0000:40:07.1: [1022:1454] type 01 class 0x060400 [ 2.293327] pci 0000:40:07.1: PME# supported from D0 D3hot D3cold [ 2.293408] pci 0000:40:08.0: [1022:1452] type 00 class 0x060000 [ 2.293474] pci 0000:40:08.1: [1022:1454] type 01 class 0x060400 [ 2.293585] pci 0000:40:08.1: PME# supported from D0 D3hot D3cold [ 2.293794] pci 0000:41:00.0: [1022:145a] type 00 class 0x130000 [ 2.293900] pci 0000:41:00.2: [1022:1456] type 00 class 0x108000 [ 2.293919] pci 0000:41:00.2: reg 0x18: [mem 0xdb300000-0xdb3fffff] [ 2.293932] pci 0000:41:00.2: reg 0x24: [mem 0xdb400000-0xdb401fff] [ 2.294017] pci 0000:41:00.3: [1022:145f] type 00 class 0x0c0330 [ 2.294030] pci 0000:41:00.3: reg 0x10: [mem 0xdb200000-0xdb2fffff 64bit] [ 2.294084] pci 0000:41:00.3: PME# supported from D0 D3hot D3cold [ 2.294145] pci 0000:40:07.1: PCI bridge to [bus 41] [ 2.299117] pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] [ 2.299327] pci 0000:42:00.0: [1022:1455] type 00 class 0x130000 [ 2.299445] pci 0000:42:00.1: [1022:1468] type 00 class 0x108000 [ 2.299466] pci 0000:42:00.1: reg 0x18: [mem 0xdb000000-0xdb0fffff] [ 2.299480] pci 0000:42:00.1: reg 0x24: [mem 0xdb100000-0xdb101fff] [ 2.299579] pci 0000:40:08.1: PCI bridge to [bus 42] [ 2.304551] pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] [ 2.304564] pci_bus 0000:40: on NUMA node 1 [ 2.304745] ACPI: PCI Root Bridge [PC02] (domain 0000 [bus 80-bf]) [ 2.310929] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.319139] acpi PNP0A08:02: PCIe AER handled by firmware [ 2.324583] acpi PNP0A08:02: _OSC: platform does not support [SHPCHotplug] [ 2.331530] acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.339180] acpi PNP0A08:02: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.347621] PCI host bridge to bus 0000:80 [ 2.351725] pci_bus 0000:80: root bus resource [io 0x03b0-0x03df window] [ 2.358508] pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff window] [ 2.365987] pci_bus 0000:80: root bus resource [io 0x8000-0xbfff window] [ 2.372775] pci_bus 0000:80: root bus resource [mem 0xab000000-0xc5ffffff window] [ 2.380254] pci_bus 0000:80: root bus resource [mem 0x47e80000000-0x63dbfffffff window] [ 2.388254] pci_bus 0000:80: root bus resource [bus 80-bf] [ 2.393745] pci 0000:80:00.0: [1022:1450] type 00 class 0x060000 [ 2.393818] pci 0000:80:00.2: [1022:1451] type 00 class 0x080600 [ 2.393906] pci 0000:80:01.0: [1022:1452] type 00 class 0x060000 [ 2.393970] pci 0000:80:01.1: [1022:1453] type 01 class 0x060400 [ 2.394355] pci 0000:80:01.1: PME# supported from D0 D3hot D3cold [ 2.394427] pci 0000:80:01.2: [1022:1453] type 01 class 0x060400 [ 2.394558] pci 0000:80:01.2: PME# supported from D0 D3hot D3cold [ 2.394642] pci 0000:80:02.0: [1022:1452] type 00 class 0x060000 [ 2.394718] pci 0000:80:03.0: [1022:1452] type 00 class 0x060000 [ 2.394778] pci 0000:80:03.1: [1022:1453] type 01 class 0x060400 [ 2.395355] pci 0000:80:03.1: PME# supported from D0 D3hot D3cold [ 2.395453] pci 0000:80:04.0: [1022:1452] type 00 class 0x060000 [ 2.395537] pci 0000:80:07.0: [1022:1452] type 00 class 0x060000 [ 2.395599] pci 0000:80:07.1: [1022:1454] type 01 class 0x060400 [ 2.395710] pci 0000:80:07.1: PME# supported from D0 D3hot D3cold [ 2.395789] pci 0000:80:08.0: [1022:1452] type 00 class 0x060000 [ 2.395851] pci 0000:80:08.1: [1022:1454] type 01 class 0x060400 [ 2.396362] pci 0000:80:08.1: PME# supported from D0 D3hot D3cold [ 2.396576] pci 0000:81:00.0: [14e4:165f] type 00 class 0x020000 [ 2.396601] pci 0000:81:00.0: reg 0x10: [mem 0xac230000-0xac23ffff 64bit pref] [ 2.396616] pci 0000:81:00.0: reg 0x18: [mem 0xac240000-0xac24ffff 64bit pref] [ 2.396631] pci 0000:81:00.0: reg 0x20: [mem 0xac250000-0xac25ffff 64bit pref] [ 2.396643] pci 0000:81:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 2.396718] pci 0000:81:00.0: PME# supported from D0 D3hot D3cold [ 2.396814] pci 0000:81:00.1: [14e4:165f] type 00 class 0x020000 [ 2.396839] pci 0000:81:00.1: reg 0x10: [mem 0xac200000-0xac20ffff 64bit pref] [ 2.396854] pci 0000:81:00.1: reg 0x18: [mem 0xac210000-0xac21ffff 64bit pref] [ 2.396869] pci 0000:81:00.1: reg 0x20: [mem 0xac220000-0xac22ffff 64bit pref] [ 2.396879] pci 0000:81:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 2.396954] pci 0000:81:00.1: PME# supported from D0 D3hot D3cold [ 2.397040] pci 0000:80:01.1: PCI bridge to [bus 81] [ 2.402016] pci 0000:80:01.1: bridge window [mem 0xac200000-0xac2fffff 64bit pref] [ 2.402353] pci 0000:82:00.0: [1556:be00] type 01 class 0x060400 [ 2.404647] pci 0000:80:01.2: PCI bridge to [bus 82-83] [ 2.409882] pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] [ 2.409887] pci 0000:80:01.2: bridge window [mem 0xab000000-0xabffffff 64bit pref] [ 2.409934] pci 0000:83:00.0: [102b:0536] type 00 class 0x030000 [ 2.409952] pci 0000:83:00.0: reg 0x10: [mem 0xab000000-0xabffffff pref] [ 2.409963] pci 0000:83:00.0: reg 0x14: [mem 0xc0808000-0xc080bfff] [ 2.409974] pci 0000:83:00.0: reg 0x18: [mem 0xc0000000-0xc07fffff] [ 2.410115] pci 0000:82:00.0: PCI bridge to [bus 83] [ 2.415085] pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] [ 2.415091] pci 0000:82:00.0: bridge window [mem 0xab000000-0xabffffff 64bit pref] [ 2.415369] pci 0000:84:00.0: [1000:00d1] type 00 class 0x010700 [ 2.415392] pci 0000:84:00.0: reg 0x10: [mem 0xac000000-0xac0fffff 64bit pref] [ 2.415402] pci 0000:84:00.0: reg 0x18: [mem 0xac100000-0xac1fffff 64bit pref] [ 2.415409] pci 0000:84:00.0: reg 0x20: [mem 0xc0d00000-0xc0dfffff] [ 2.415417] pci 0000:84:00.0: reg 0x24: [io 0x8000-0x80ff] [ 2.415425] pci 0000:84:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref] [ 2.415478] pci 0000:84:00.0: supports D1 D2 [ 2.417645] pci 0000:80:03.1: PCI bridge to [bus 84] [ 2.422612] pci 0000:80:03.1: bridge window [io 0x8000-0x8fff] [ 2.422614] pci 0000:80:03.1: bridge window [mem 0xc0d00000-0xc0dfffff] [ 2.422618] pci 0000:80:03.1: bridge window [mem 0xac000000-0xac1fffff 64bit pref] [ 2.422697] pci 0000:85:00.0: [1022:145a] type 00 class 0x130000 [ 2.422803] pci 0000:85:00.2: [1022:1456] type 00 class 0x108000 [ 2.422821] pci 0000:85:00.2: reg 0x18: [mem 0xc0b00000-0xc0bfffff] [ 2.422834] pci 0000:85:00.2: reg 0x24: [mem 0xc0c00000-0xc0c01fff] [ 2.422927] pci 0000:80:07.1: PCI bridge to [bus 85] [ 2.427900] pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] [ 2.428413] pci 0000:86:00.0: [1022:1455] type 00 class 0x130000 [ 2.428532] pci 0000:86:00.1: [1022:1468] type 00 class 0x108000 [ 2.428552] pci 0000:86:00.1: reg 0x18: [mem 0xc0900000-0xc09fffff] [ 2.428566] pci 0000:86:00.1: reg 0x24: [mem 0xc0a00000-0xc0a01fff] [ 2.428655] pci 0000:86:00.2: [1022:7901] type 00 class 0x010601 [ 2.428687] pci 0000:86:00.2: reg 0x24: [mem 0xc0a02000-0xc0a02fff] [ 2.428726] pci 0000:86:00.2: PME# supported from D3hot D3cold [ 2.428793] pci 0000:80:08.1: PCI bridge to [bus 86] [ 2.433766] pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] [ 2.433791] pci_bus 0000:80: on NUMA node 2 [ 2.433960] ACPI: PCI Root Bridge [PC03] (domain 0000 [bus c0-ff]) [ 2.440145] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.448353] acpi PNP0A08:03: PCIe AER handled by firmware [ 2.453797] acpi PNP0A08:03: _OSC: platform does not support [SHPCHotplug] [ 2.460745] acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.468395] acpi PNP0A08:03: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.476722] acpi PNP0A08:03: host bridge window [mem 0x63dc0000000-0xffffffffffff window] ([0x80000000000-0xffffffffffff] ignored, not CPU addressable) [ 2.490361] PCI host bridge to bus 0000:c0 [ 2.494459] pci_bus 0000:c0: root bus resource [io 0xc000-0xffff window] [ 2.501244] pci_bus 0000:c0: root bus resource [mem 0x90000000-0xaaffffff window] [ 2.508724] pci_bus 0000:c0: root bus resource [mem 0x63dc0000000-0x7ffffffffff window] [ 2.516724] pci_bus 0000:c0: root bus resource [bus c0-ff] [ 2.522213] pci 0000:c0:00.0: [1022:1450] type 00 class 0x060000 [ 2.522283] pci 0000:c0:00.2: [1022:1451] type 00 class 0x080600 [ 2.522374] pci 0000:c0:01.0: [1022:1452] type 00 class 0x060000 [ 2.522435] pci 0000:c0:01.1: [1022:1453] type 01 class 0x060400 [ 2.522615] pci 0000:c0:01.1: PME# supported from D0 D3hot D3cold [ 2.522714] pci 0000:c0:02.0: [1022:1452] type 00 class 0x060000 [ 2.522789] pci 0000:c0:03.0: [1022:1452] type 00 class 0x060000 [ 2.522864] pci 0000:c0:04.0: [1022:1452] type 00 class 0x060000 [ 2.522943] pci 0000:c0:07.0: [1022:1452] type 00 class 0x060000 [ 2.523004] pci 0000:c0:07.1: [1022:1454] type 01 class 0x060400 [ 2.523595] pci 0000:c0:07.1: PME# supported from D0 D3hot D3cold [ 2.523675] pci 0000:c0:08.0: [1022:1452] type 00 class 0x060000 [ 2.523738] pci 0000:c0:08.1: [1022:1454] type 01 class 0x060400 [ 2.523850] pci 0000:c0:08.1: PME# supported from D0 D3hot D3cold [ 2.524049] pci 0000:c1:00.0: [1000:005f] type 00 class 0x010400 [ 2.524063] pci 0000:c1:00.0: reg 0x10: [io 0xc000-0xc0ff] [ 2.524073] pci 0000:c1:00.0: reg 0x14: [mem 0xa5500000-0xa550ffff 64bit] [ 2.524083] pci 0000:c1:00.0: reg 0x1c: [mem 0xa5400000-0xa54fffff 64bit] [ 2.524095] pci 0000:c1:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 2.524143] pci 0000:c1:00.0: supports D1 D2 [ 2.524194] pci 0000:c0:01.1: PCI bridge to [bus c1] [ 2.529162] pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] [ 2.529164] pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] [ 2.529601] pci 0000:c2:00.0: [1022:145a] type 00 class 0x130000 [ 2.529708] pci 0000:c2:00.2: [1022:1456] type 00 class 0x108000 [ 2.529727] pci 0000:c2:00.2: reg 0x18: [mem 0xa5200000-0xa52fffff] [ 2.529740] pci 0000:c2:00.2: reg 0x24: [mem 0xa5300000-0xa5301fff] [ 2.529833] pci 0000:c0:07.1: PCI bridge to [bus c2] [ 2.534806] pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] [ 2.534900] pci 0000:c3:00.0: [1022:1455] type 00 class 0x130000 [ 2.535016] pci 0000:c3:00.1: [1022:1468] type 00 class 0x108000 [ 2.535035] pci 0000:c3:00.1: reg 0x18: [mem 0xa5000000-0xa50fffff] [ 2.535050] pci 0000:c3:00.1: reg 0x24: [mem 0xa5100000-0xa5101fff] [ 2.535148] pci 0000:c0:08.1: PCI bridge to [bus c3] [ 2.540119] pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] [ 2.540136] pci_bus 0000:c0: on NUMA node 3 [ 2.542288] vgaarb: device added: PCI:0000:83:00.0,decodes=io+mem,owns=io+mem,locks=none [ 2.550380] vgaarb: loaded [ 2.553089] vgaarb: bridge control possible 0000:83:00.0 [ 2.558516] SCSI subsystem initialized [ 2.562293] ACPI: bus type USB registered [ 2.566322] usbcore: registered new interface driver usbfs [ 2.571816] usbcore: registered new interface driver hub [ 2.577341] usbcore: registered new device driver usb [ 2.582716] EDAC MC: Ver: 3.0.0 [ 2.586111] PCI: Using ACPI for IRQ routing [ 2.609269] PCI: pci_cache_line_size set to 64 bytes [ 2.609421] e820: reserve RAM buffer [mem 0x0008f000-0x0008ffff] [ 2.609423] e820: reserve RAM buffer [mem 0x378b2020-0x37ffffff] [ 2.609425] e820: reserve RAM buffer [mem 0x378cb020-0x37ffffff] [ 2.609427] e820: reserve RAM buffer [mem 0x378f1020-0x37ffffff] [ 2.609428] e820: reserve RAM buffer [mem 0x378fa020-0x37ffffff] [ 2.609430] e820: reserve RAM buffer [mem 0x3792c020-0x37ffffff] [ 2.609431] e820: reserve RAM buffer [mem 0x3795e020-0x37ffffff] [ 2.609432] e820: reserve RAM buffer [mem 0x4f781000-0x4fffffff] [ 2.609433] e820: reserve RAM buffer [mem 0x6cacf000-0x6fffffff] [ 2.609435] e820: reserve RAM buffer [mem 0x107f380000-0x107fffffff] [ 2.609436] e820: reserve RAM buffer [mem 0x207ff80000-0x207fffffff] [ 2.609437] e820: reserve RAM buffer [mem 0x307ff80000-0x307fffffff] [ 2.609439] e820: reserve RAM buffer [mem 0x407ff80000-0x407fffffff] [ 2.609693] NetLabel: Initializing [ 2.613099] NetLabel: domain hash size = 128 [ 2.617457] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.622440] NetLabel: unlabeled traffic allowed by default [ 2.628211] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 2.633196] hpet0: 3 comparators, 32-bit 14.318180 MHz counter [ 2.641208] Switched to clocksource hpet [ 2.649877] pnp: PnP ACPI init [ 2.652959] ACPI: bus type PNP registered [ 2.657149] system 00:00: [mem 0x80000000-0x8fffffff] has been reserved [ 2.663775] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 2.663831] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active) [ 2.664025] pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.664211] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.664352] pnp: PnP ACPI: found 4 devices [ 2.668462] ACPI: bus type PNP unregistered [ 2.679923] pci 0000:01:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.689840] pci 0000:81:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.699759] pci 0000:81:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.709677] pci 0000:c1:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.719612] pci 0000:00:03.1: BAR 14: assigned [mem 0xe1000000-0xe10fffff] [ 2.726499] pci 0000:01:00.0: BAR 6: assigned [mem 0xe1000000-0xe10fffff pref] [ 2.733726] pci 0000:00:03.1: PCI bridge to [bus 01] [ 2.738701] pci 0000:00:03.1: bridge window [mem 0xe1000000-0xe10fffff] [ 2.745495] pci 0000:00:03.1: bridge window [mem 0xe2000000-0xe3ffffff 64bit pref] [ 2.753247] pci 0000:00:07.1: PCI bridge to [bus 02] [ 2.758219] pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] [ 2.765015] pci 0000:00:08.1: PCI bridge to [bus 03] [ 2.769989] pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] [ 2.776787] pci_bus 0000:00: resource 4 [io 0x0000-0x03af window] [ 2.776789] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7 window] [ 2.776791] pci_bus 0000:00: resource 6 [mem 0x000c0000-0x000c3fff window] [ 2.776792] pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff window] [ 2.776794] pci_bus 0000:00: resource 8 [mem 0x000c8000-0x000cbfff window] [ 2.776796] pci_bus 0000:00: resource 9 [mem 0x000cc000-0x000cffff window] [ 2.776798] pci_bus 0000:00: resource 10 [mem 0x000d0000-0x000d3fff window] [ 2.776799] pci_bus 0000:00: resource 11 [mem 0x000d4000-0x000d7fff window] [ 2.776801] pci_bus 0000:00: resource 12 [mem 0x000d8000-0x000dbfff window] [ 2.776803] pci_bus 0000:00: resource 13 [mem 0x000dc000-0x000dffff window] [ 2.776804] pci_bus 0000:00: resource 14 [mem 0x000e0000-0x000e3fff window] [ 2.776806] pci_bus 0000:00: resource 15 [mem 0x000e4000-0x000e7fff window] [ 2.776808] pci_bus 0000:00: resource 16 [mem 0x000e8000-0x000ebfff window] [ 2.776809] pci_bus 0000:00: resource 17 [mem 0x000ec000-0x000effff window] [ 2.776811] pci_bus 0000:00: resource 18 [mem 0x000f0000-0x000fffff window] [ 2.776813] pci_bus 0000:00: resource 19 [io 0x0d00-0x3fff window] [ 2.776815] pci_bus 0000:00: resource 20 [mem 0xe1000000-0xfebfffff window] [ 2.776816] pci_bus 0000:00: resource 21 [mem 0x10000000000-0x2bf3fffffff window] [ 2.776818] pci_bus 0000:01: resource 1 [mem 0xe1000000-0xe10fffff] [ 2.776820] pci_bus 0000:01: resource 2 [mem 0xe2000000-0xe3ffffff 64bit pref] [ 2.776822] pci_bus 0000:02: resource 1 [mem 0xf7200000-0xf74fffff] [ 2.776824] pci_bus 0000:03: resource 1 [mem 0xf7000000-0xf71fffff] [ 2.776836] pci 0000:40:07.1: PCI bridge to [bus 41] [ 2.781811] pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] [ 2.788606] pci 0000:40:08.1: PCI bridge to [bus 42] [ 2.793581] pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] [ 2.800377] pci_bus 0000:40: resource 4 [io 0x4000-0x7fff window] [ 2.800379] pci_bus 0000:40: resource 5 [mem 0xc6000000-0xe0ffffff window] [ 2.800381] pci_bus 0000:40: resource 6 [mem 0x2bf40000000-0x47e7fffffff window] [ 2.800382] pci_bus 0000:41: resource 1 [mem 0xdb200000-0xdb4fffff] [ 2.800384] pci_bus 0000:42: resource 1 [mem 0xdb000000-0xdb1fffff] [ 2.800415] pci 0000:80:01.1: BAR 14: assigned [mem 0xac300000-0xac3fffff] [ 2.807301] pci 0000:81:00.0: BAR 6: assigned [mem 0xac300000-0xac33ffff pref] [ 2.814527] pci 0000:81:00.1: BAR 6: assigned [mem 0xac340000-0xac37ffff pref] [ 2.821754] pci 0000:80:01.1: PCI bridge to [bus 81] [ 2.826731] pci 0000:80:01.1: bridge window [mem 0xac300000-0xac3fffff] [ 2.833525] pci 0000:80:01.1: bridge window [mem 0xac200000-0xac2fffff 64bit pref] [ 2.841276] pci 0000:82:00.0: PCI bridge to [bus 83] [ 2.846250] pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] [ 2.853044] pci 0000:82:00.0: bridge window [mem 0xab000000-0xabffffff 64bit pref] [ 2.860795] pci 0000:80:01.2: PCI bridge to [bus 82-83] [ 2.866027] pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] [ 2.872820] pci 0000:80:01.2: bridge window [mem 0xab000000-0xabffffff 64bit pref] [ 2.880572] pci 0000:84:00.0: BAR 6: no space for [mem size 0x00040000 pref] [ 2.887624] pci 0000:84:00.0: BAR 6: failed to assign [mem size 0x00040000 pref] [ 2.895025] pci 0000:80:03.1: PCI bridge to [bus 84] [ 2.900001] pci 0000:80:03.1: bridge window [io 0x8000-0x8fff] [ 2.906103] pci 0000:80:03.1: bridge window [mem 0xc0d00000-0xc0dfffff] [ 2.912897] pci 0000:80:03.1: bridge window [mem 0xac000000-0xac1fffff 64bit pref] [ 2.920645] pci 0000:80:07.1: PCI bridge to [bus 85] [ 2.925620] pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] [ 2.932417] pci 0000:80:08.1: PCI bridge to [bus 86] [ 2.937389] pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] [ 2.944187] pci_bus 0000:80: resource 4 [io 0x03b0-0x03df window] [ 2.944189] pci_bus 0000:80: resource 5 [mem 0x000a0000-0x000bffff window] [ 2.944191] pci_bus 0000:80: resource 6 [io 0x8000-0xbfff window] [ 2.944192] pci_bus 0000:80: resource 7 [mem 0xab000000-0xc5ffffff window] [ 2.944194] pci_bus 0000:80: resource 8 [mem 0x47e80000000-0x63dbfffffff window] [ 2.944205] pci_bus 0000:81: resource 1 [mem 0xac300000-0xac3fffff] [ 2.944207] pci_bus 0000:81: resource 2 [mem 0xac200000-0xac2fffff 64bit pref] [ 2.944209] pci_bus 0000:82: resource 1 [mem 0xc0000000-0xc08fffff] [ 2.944210] pci_bus 0000:82: resource 2 [mem 0xab000000-0xabffffff 64bit pref] [ 2.944212] pci_bus 0000:83: resource 1 [mem 0xc0000000-0xc08fffff] [ 2.944214] pci_bus 0000:83: resource 2 [mem 0xab000000-0xabffffff 64bit pref] [ 2.944216] pci_bus 0000:84: resource 0 [io 0x8000-0x8fff] [ 2.944217] pci_bus 0000:84: resource 1 [mem 0xc0d00000-0xc0dfffff] [ 2.944219] pci_bus 0000:84: resource 2 [mem 0xac000000-0xac1fffff 64bit pref] [ 2.944221] pci_bus 0000:85: resource 1 [mem 0xc0b00000-0xc0cfffff] [ 2.944223] pci_bus 0000:86: resource 1 [mem 0xc0900000-0xc0afffff] [ 2.944238] pci 0000:c1:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 2.951290] pci 0000:c1:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 2.958693] pci 0000:c0:01.1: PCI bridge to [bus c1] [ 2.963667] pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] [ 2.969769] pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] [ 2.976568] pci 0000:c0:07.1: PCI bridge to [bus c2] [ 2.981547] pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] [ 2.988346] pci 0000:c0:08.1: PCI bridge to [bus c3] [ 2.993326] pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] [ 3.000124] pci_bus 0000:c0: resource 4 [io 0xc000-0xffff window] [ 3.000126] pci_bus 0000:c0: resource 5 [mem 0x90000000-0xaaffffff window] [ 3.000128] pci_bus 0000:c0: resource 6 [mem 0x63dc0000000-0x7ffffffffff window] [ 3.000129] pci_bus 0000:c1: resource 0 [io 0xc000-0xcfff] [ 3.000131] pci_bus 0000:c1: resource 1 [mem 0xa5400000-0xa55fffff] [ 3.000133] pci_bus 0000:c2: resource 1 [mem 0xa5200000-0xa53fffff] [ 3.000134] pci_bus 0000:c3: resource 1 [mem 0xa5000000-0xa51fffff] [ 3.000227] NET: Registered protocol family 2 [ 3.005274] TCP established hash table entries: 524288 (order: 10, 4194304 bytes) [ 3.013423] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 3.020260] TCP: Hash tables configured (established 524288 bind 65536) [ 3.026917] TCP: reno registered [ 3.030281] UDP hash table entries: 65536 (order: 9, 2097152 bytes) [ 3.036881] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) [ 3.044089] NET: Registered protocol family 1 [ 3.048894] pci 0000:83:00.0: Boot video device [ 3.048933] PCI: CLS 64 bytes, default 64 [ 3.048993] Unpacking initramfs... [ 3.318530] Freeing initrd memory: 19596k freed [ 3.325232] AMD-Vi: IOMMU performance counters supported [ 3.330617] AMD-Vi: IOMMU performance counters supported [ 3.335969] AMD-Vi: IOMMU performance counters supported [ 3.341324] AMD-Vi: IOMMU performance counters supported [ 3.347953] iommu: Adding device 0000:00:01.0 to group 0 [ 3.353977] iommu: Adding device 0000:00:02.0 to group 1 [ 3.360023] iommu: Adding device 0000:00:03.0 to group 2 [ 3.366116] iommu: Adding device 0000:00:03.1 to group 3 [ 3.372105] iommu: Adding device 0000:00:04.0 to group 4 [ 3.378110] iommu: Adding device 0000:00:07.0 to group 5 [ 3.384126] iommu: Adding device 0000:00:07.1 to group 6 [ 3.390140] iommu: Adding device 0000:00:08.0 to group 7 [ 3.396100] iommu: Adding device 0000:00:08.1 to group 8 [ 3.402133] iommu: Adding device 0000:00:14.0 to group 9 [ 3.407470] iommu: Adding device 0000:00:14.3 to group 9 [ 3.413578] iommu: Adding device 0000:00:18.0 to group 10 [ 3.419002] iommu: Adding device 0000:00:18.1 to group 10 [ 3.424427] iommu: Adding device 0000:00:18.2 to group 10 [ 3.429851] iommu: Adding device 0000:00:18.3 to group 10 [ 3.435277] iommu: Adding device 0000:00:18.4 to group 10 [ 3.440700] iommu: Adding device 0000:00:18.5 to group 10 [ 3.446132] iommu: Adding device 0000:00:18.6 to group 10 [ 3.451563] iommu: Adding device 0000:00:18.7 to group 10 [ 3.457720] iommu: Adding device 0000:00:19.0 to group 11 [ 3.463150] iommu: Adding device 0000:00:19.1 to group 11 [ 3.468576] iommu: Adding device 0000:00:19.2 to group 11 [ 3.473998] iommu: Adding device 0000:00:19.3 to group 11 [ 3.479423] iommu: Adding device 0000:00:19.4 to group 11 [ 3.484853] iommu: Adding device 0000:00:19.5 to group 11 [ 3.490279] iommu: Adding device 0000:00:19.6 to group 11 [ 3.495704] iommu: Adding device 0000:00:19.7 to group 11 [ 3.501873] iommu: Adding device 0000:00:1a.0 to group 12 [ 3.507300] iommu: Adding device 0000:00:1a.1 to group 12 [ 3.512724] iommu: Adding device 0000:00:1a.2 to group 12 [ 3.518153] iommu: Adding device 0000:00:1a.3 to group 12 [ 3.523576] iommu: Adding device 0000:00:1a.4 to group 12 [ 3.529002] iommu: Adding device 0000:00:1a.5 to group 12 [ 3.534430] iommu: Adding device 0000:00:1a.6 to group 12 [ 3.539851] iommu: Adding device 0000:00:1a.7 to group 12 [ 3.545992] iommu: Adding device 0000:00:1b.0 to group 13 [ 3.551414] iommu: Adding device 0000:00:1b.1 to group 13 [ 3.556841] iommu: Adding device 0000:00:1b.2 to group 13 [ 3.562266] iommu: Adding device 0000:00:1b.3 to group 13 [ 3.567693] iommu: Adding device 0000:00:1b.4 to group 13 [ 3.573116] iommu: Adding device 0000:00:1b.5 to group 13 [ 3.578542] iommu: Adding device 0000:00:1b.6 to group 13 [ 3.583968] iommu: Adding device 0000:00:1b.7 to group 13 [ 3.590124] iommu: Adding device 0000:01:00.0 to group 14 [ 3.596227] iommu: Adding device 0000:02:00.0 to group 15 [ 3.602334] iommu: Adding device 0000:02:00.2 to group 16 [ 3.608454] iommu: Adding device 0000:02:00.3 to group 17 [ 3.614542] iommu: Adding device 0000:03:00.0 to group 18 [ 3.620620] iommu: Adding device 0000:03:00.1 to group 19 [ 3.626743] iommu: Adding device 0000:40:01.0 to group 20 [ 3.632863] iommu: Adding device 0000:40:02.0 to group 21 [ 3.638979] iommu: Adding device 0000:40:03.0 to group 22 [ 3.645090] iommu: Adding device 0000:40:04.0 to group 23 [ 3.651175] iommu: Adding device 0000:40:07.0 to group 24 [ 3.657245] iommu: Adding device 0000:40:07.1 to group 25 [ 3.663281] iommu: Adding device 0000:40:08.0 to group 26 [ 3.669349] iommu: Adding device 0000:40:08.1 to group 27 [ 3.675383] iommu: Adding device 0000:41:00.0 to group 28 [ 3.681391] iommu: Adding device 0000:41:00.2 to group 29 [ 3.687428] iommu: Adding device 0000:41:00.3 to group 30 [ 3.693463] iommu: Adding device 0000:42:00.0 to group 31 [ 3.699531] iommu: Adding device 0000:42:00.1 to group 32 [ 3.705563] iommu: Adding device 0000:80:01.0 to group 33 [ 3.711562] iommu: Adding device 0000:80:01.1 to group 34 [ 3.717723] iommu: Adding device 0000:80:01.2 to group 35 [ 3.723783] iommu: Adding device 0000:80:02.0 to group 36 [ 3.729814] iommu: Adding device 0000:80:03.0 to group 37 [ 3.735847] iommu: Adding device 0000:80:03.1 to group 38 [ 3.741873] iommu: Adding device 0000:80:04.0 to group 39 [ 3.747934] iommu: Adding device 0000:80:07.0 to group 40 [ 3.753970] iommu: Adding device 0000:80:07.1 to group 41 [ 3.760011] iommu: Adding device 0000:80:08.0 to group 42 [ 3.766079] iommu: Adding device 0000:80:08.1 to group 43 [ 3.772150] iommu: Adding device 0000:81:00.0 to group 44 [ 3.777600] iommu: Adding device 0000:81:00.1 to group 44 [ 3.783625] iommu: Adding device 0000:82:00.0 to group 45 [ 3.789041] iommu: Adding device 0000:83:00.0 to group 45 [ 3.795105] iommu: Adding device 0000:84:00.0 to group 46 [ 3.801158] iommu: Adding device 0000:85:00.0 to group 47 [ 3.807247] iommu: Adding device 0000:85:00.2 to group 48 [ 3.813296] iommu: Adding device 0000:86:00.0 to group 49 [ 3.819314] iommu: Adding device 0000:86:00.1 to group 50 [ 3.825355] iommu: Adding device 0000:86:00.2 to group 51 [ 3.831418] iommu: Adding device 0000:c0:01.0 to group 52 [ 3.837438] iommu: Adding device 0000:c0:01.1 to group 53 [ 3.843490] iommu: Adding device 0000:c0:02.0 to group 54 [ 3.849538] iommu: Adding device 0000:c0:03.0 to group 55 [ 3.855632] iommu: Adding device 0000:c0:04.0 to group 56 [ 3.861642] iommu: Adding device 0000:c0:07.0 to group 57 [ 3.867714] iommu: Adding device 0000:c0:07.1 to group 58 [ 3.873796] iommu: Adding device 0000:c0:08.0 to group 59 [ 3.879802] iommu: Adding device 0000:c0:08.1 to group 60 [ 3.888284] iommu: Adding device 0000:c1:00.0 to group 61 [ 3.894370] iommu: Adding device 0000:c2:00.0 to group 62 [ 3.900435] iommu: Adding device 0000:c2:00.2 to group 63 [ 3.906455] iommu: Adding device 0000:c3:00.0 to group 64 [ 3.912507] iommu: Adding device 0000:c3:00.1 to group 65 [ 3.918117] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 3.923441] AMD-Vi: Extended features (0xf77ef22294ada): [ 3.928762] PPR NX GT IA GA PC GA_vAPIC [ 3.932896] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 [ 3.938217] AMD-Vi: Extended features (0xf77ef22294ada): [ 3.943537] PPR NX GT IA GA PC GA_vAPIC [ 3.947680] AMD-Vi: Found IOMMU at 0000:80:00.2 cap 0x40 [ 3.953004] AMD-Vi: Extended features (0xf77ef22294ada): [ 3.958324] PPR NX GT IA GA PC GA_vAPIC [ 3.962465] AMD-Vi: Found IOMMU at 0000:c0:00.2 cap 0x40 [ 3.967789] AMD-Vi: Extended features (0xf77ef22294ada): [ 3.973112] PPR NX GT IA GA PC GA_vAPIC [ 3.977253] AMD-Vi: Interrupt remapping enabled [ 3.981795] AMD-Vi: virtual APIC enabled [ 3.985785] pci 0000:00:00.2: irq 26 for MSI/MSI-X [ 3.985883] pci 0000:40:00.2: irq 27 for MSI/MSI-X [ 3.985965] pci 0000:80:00.2: irq 28 for MSI/MSI-X [ 3.986046] pci 0000:c0:00.2: irq 29 for MSI/MSI-X [ 3.986099] AMD-Vi: Lazy IO/TLB flushing enabled [ 3.992437] perf: AMD NB counters detected [ 3.996584] perf: AMD LLC counters detected [ 4.006755] sha1_ssse3: Using SHA-NI optimized SHA-1 implementation [ 4.013107] sha256_ssse3: Using SHA-256-NI optimized SHA-256 implementation [ 4.021684] futex hash table entries: 32768 (order: 9, 2097152 bytes) [ 4.028320] Initialise system trusted keyring [ 4.032721] audit: initializing netlink socket (disabled) [ 4.038140] type=2000 audit(1584554573.187:1): initialized [ 4.068977] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.075337] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.082957] zpool: loaded [ 4.085589] zbud: loaded [ 4.088500] VFS: Disk quotas dquot_6.6.0 [ 4.092533] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.099349] msgmni has been set to 32768 [ 4.103381] Key type big_key registered [ 4.107231] SELinux: Registering netfilter hooks [ 4.109661] NET: Registered protocol family 38 [ 4.114117] Key type asymmetric registered [ 4.118221] Asymmetric key parser 'x509' registered [ 4.123157] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 4.130710] io scheduler noop registered [ 4.134644] io scheduler deadline registered (default) [ 4.139827] io scheduler cfq registered [ 4.143674] io scheduler mq-deadline registered [ 4.148216] io scheduler kyber registered [ 4.153083] pcieport 0000:00:03.1: irq 30 for MSI/MSI-X [ 4.153983] pcieport 0000:00:07.1: irq 31 for MSI/MSI-X [ 4.154219] pcieport 0000:00:08.1: irq 33 for MSI/MSI-X [ 4.155189] pcieport 0000:40:07.1: irq 34 for MSI/MSI-X [ 4.155889] pcieport 0000:40:08.1: irq 36 for MSI/MSI-X [ 4.156669] pcieport 0000:80:01.1: irq 37 for MSI/MSI-X [ 4.156921] pcieport 0000:80:01.2: irq 38 for MSI/MSI-X [ 4.157134] pcieport 0000:80:03.1: irq 39 for MSI/MSI-X [ 4.157927] pcieport 0000:80:07.1: irq 41 for MSI/MSI-X [ 4.158639] pcieport 0000:80:08.1: irq 43 for MSI/MSI-X [ 4.158974] pcieport 0000:c0:01.1: irq 44 for MSI/MSI-X [ 4.159766] pcieport 0000:c0:07.1: irq 46 for MSI/MSI-X [ 4.160004] pcieport 0000:c0:08.1: irq 48 for MSI/MSI-X [ 4.160117] pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt [ 4.167084] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt [ 4.173620] pcie_pme 0000:00:03.1:pcie001: service driver pcie_pme loaded [ 4.173631] pcieport 0000:00:07.1: Signaling PME through PCIe PME interrupt [ 4.180595] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt [ 4.187128] pci 0000:02:00.2: Signaling PME through PCIe PME interrupt [ 4.193663] pci 0000:02:00.3: Signaling PME through PCIe PME interrupt [ 4.200200] pcie_pme 0000:00:07.1:pcie001: service driver pcie_pme loaded [ 4.200212] pcieport 0000:00:08.1: Signaling PME through PCIe PME interrupt [ 4.207177] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt [ 4.213711] pci 0000:03:00.1: Signaling PME through PCIe PME interrupt [ 4.220246] pcie_pme 0000:00:08.1:pcie001: service driver pcie_pme loaded [ 4.220265] pcieport 0000:40:07.1: Signaling PME through PCIe PME interrupt [ 4.227232] pci 0000:41:00.0: Signaling PME through PCIe PME interrupt [ 4.233764] pci 0000:41:00.2: Signaling PME through PCIe PME interrupt [ 4.240300] pci 0000:41:00.3: Signaling PME through PCIe PME interrupt [ 4.246836] pcie_pme 0000:40:07.1:pcie001: service driver pcie_pme loaded [ 4.246851] pcieport 0000:40:08.1: Signaling PME through PCIe PME interrupt [ 4.253822] pci 0000:42:00.0: Signaling PME through PCIe PME interrupt [ 4.260354] pci 0000:42:00.1: Signaling PME through PCIe PME interrupt [ 4.266893] pcie_pme 0000:40:08.1:pcie001: service driver pcie_pme loaded [ 4.266912] pcieport 0000:80:01.1: Signaling PME through PCIe PME interrupt [ 4.273877] pci 0000:81:00.0: Signaling PME through PCIe PME interrupt [ 4.280411] pci 0000:81:00.1: Signaling PME through PCIe PME interrupt [ 4.286949] pcie_pme 0000:80:01.1:pcie001: service driver pcie_pme loaded [ 4.286964] pcieport 0000:80:01.2: Signaling PME through PCIe PME interrupt [ 4.293932] pci 0000:82:00.0: Signaling PME through PCIe PME interrupt [ 4.300465] pci 0000:83:00.0: Signaling PME through PCIe PME interrupt [ 4.307002] pcie_pme 0000:80:01.2:pcie001: service driver pcie_pme loaded [ 4.307017] pcieport 0000:80:03.1: Signaling PME through PCIe PME interrupt [ 4.313987] pci 0000:84:00.0: Signaling PME through PCIe PME interrupt [ 4.320524] pcie_pme 0000:80:03.1:pcie001: service driver pcie_pme loaded [ 4.320539] pcieport 0000:80:07.1: Signaling PME through PCIe PME interrupt [ 4.327508] pci 0000:85:00.0: Signaling PME through PCIe PME interrupt [ 4.334041] pci 0000:85:00.2: Signaling PME through PCIe PME interrupt [ 4.340578] pcie_pme 0000:80:07.1:pcie001: service driver pcie_pme loaded [ 4.340592] pcieport 0000:80:08.1: Signaling PME through PCIe PME interrupt [ 4.347563] pci 0000:86:00.0: Signaling PME through PCIe PME interrupt [ 4.354097] pci 0000:86:00.1: Signaling PME through PCIe PME interrupt [ 4.360632] pci 0000:86:00.2: Signaling PME through PCIe PME interrupt [ 4.367169] pcie_pme 0000:80:08.1:pcie001: service driver pcie_pme loaded [ 4.367184] pcieport 0000:c0:01.1: Signaling PME through PCIe PME interrupt [ 4.374152] pci 0000:c1:00.0: Signaling PME through PCIe PME interrupt [ 4.380689] pcie_pme 0000:c0:01.1:pcie001: service driver pcie_pme loaded [ 4.380706] pcieport 0000:c0:07.1: Signaling PME through PCIe PME interrupt [ 4.387672] pci 0000:c2:00.0: Signaling PME through PCIe PME interrupt [ 4.394208] pci 0000:c2:00.2: Signaling PME through PCIe PME interrupt [ 4.400744] pcie_pme 0000:c0:07.1:pcie001: service driver pcie_pme loaded [ 4.400758] pcieport 0000:c0:08.1: Signaling PME through PCIe PME interrupt [ 4.407729] pci 0000:c3:00.0: Signaling PME through PCIe PME interrupt [ 4.414263] pci 0000:c3:00.1: Signaling PME through PCIe PME interrupt [ 4.420799] pcie_pme 0000:c0:08.1:pcie001: service driver pcie_pme loaded [ 4.420819] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.426401] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.433073] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.439880] efifb: probing for efifb [ 4.443473] efifb: framebuffer at 0xab000000, mapped to 0xffffb8d199800000, using 3072k, total 3072k [ 4.452605] efifb: mode is 1024x768x32, linelength=4096, pages=1 [ 4.458619] efifb: scrolling: redraw [ 4.462208] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0 [ 4.483530] Console: switching to colour frame buffer device 128x48 [ 4.505337] fb0: EFI VGA frame buffer device [ 4.509718] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0 [ 4.517905] ACPI: Power Button [PWRB] [ 4.521626] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 4.529021] ACPI: Power Button [PWRF] [ 4.533892] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 4.541380] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.568582] 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 4.595119] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.601175] Non-volatile memory driver v1.3 [ 4.605408] Linux agpgart interface v0.103 [ 4.611148] crash memory driver: version 1.1 [ 4.615658] rdac: device handler registered [ 4.619902] hp_sw: device handler registered [ 4.624188] emc: device handler registered [ 4.628442] alua: device handler registered [ 4.632672] libphy: Fixed MDIO Bus: probed [ 4.636832] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.643371] ehci-pci: EHCI PCI platform driver [ 4.647837] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.654028] ohci-pci: OHCI PCI platform driver [ 4.658494] uhci_hcd: USB Universal Host Controller Interface driver [ 4.664964] xhci_hcd 0000:02:00.3: xHCI Host Controller [ 4.670261] xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 1 [ 4.677772] xhci_hcd 0000:02:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 [ 4.686260] xhci_hcd 0000:02:00.3: irq 50 for MSI/MSI-X [ 4.686284] xhci_hcd 0000:02:00.3: irq 51 for MSI/MSI-X [ 4.686302] xhci_hcd 0000:02:00.3: irq 52 for MSI/MSI-X [ 4.686322] xhci_hcd 0000:02:00.3: irq 53 for MSI/MSI-X [ 4.686341] xhci_hcd 0000:02:00.3: irq 54 for MSI/MSI-X [ 4.686359] xhci_hcd 0000:02:00.3: irq 55 for MSI/MSI-X [ 4.686377] xhci_hcd 0000:02:00.3: irq 56 for MSI/MSI-X [ 4.686396] xhci_hcd 0000:02:00.3: irq 57 for MSI/MSI-X [ 4.686529] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 4.693322] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.700551] usb usb1: Product: xHCI Host Controller [ 4.705438] usb usb1: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd [ 4.713532] usb usb1: SerialNumber: 0000:02:00.3 [ 4.718281] hub 1-0:1.0: USB hub found [ 4.722049] hub 1-0:1.0: 2 ports detected [ 4.726308] xhci_hcd 0000:02:00.3: xHCI Host Controller [ 4.731609] xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 2 [ 4.739026] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. [ 4.747130] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 [ 4.753922] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.761151] usb usb2: Product: xHCI Host Controller [ 4.766038] usb usb2: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd [ 4.774132] usb usb2: SerialNumber: 0000:02:00.3 [ 4.778862] hub 2-0:1.0: USB hub found [ 4.782629] hub 2-0:1.0: 2 ports detected [ 4.786946] xhci_hcd 0000:41:00.3: xHCI Host Controller [ 4.792260] xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 3 [ 4.799769] xhci_hcd 0000:41:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 [ 4.808266] xhci_hcd 0000:41:00.3: irq 59 for MSI/MSI-X [ 4.808286] xhci_hcd 0000:41:00.3: irq 60 for MSI/MSI-X [ 4.808306] xhci_hcd 0000:41:00.3: irq 61 for MSI/MSI-X [ 4.808325] xhci_hcd 0000:41:00.3: irq 62 for MSI/MSI-X [ 4.808345] xhci_hcd 0000:41:00.3: irq 63 for MSI/MSI-X [ 4.808370] xhci_hcd 0000:41:00.3: irq 64 for MSI/MSI-X [ 4.808388] xhci_hcd 0000:41:00.3: irq 65 for MSI/MSI-X [ 4.808406] xhci_hcd 0000:41:00.3: irq 66 for MSI/MSI-X [ 4.808561] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 [ 4.815354] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.822582] usb usb3: Product: xHCI Host Controller [ 4.827470] usb usb3: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd [ 4.835565] usb usb3: SerialNumber: 0000:41:00.3 [ 4.840312] hub 3-0:1.0: USB hub found [ 4.844079] hub 3-0:1.0: 2 ports detected [ 4.848353] xhci_hcd 0000:41:00.3: xHCI Host Controller [ 4.853630] xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 4 [ 4.861079] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM. [ 4.869188] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003 [ 4.875988] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.883214] usb usb4: Product: xHCI Host Controller [ 4.888102] usb usb4: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd [ 4.896198] usb usb4: SerialNumber: 0000:41:00.3 [ 4.900913] hub 4-0:1.0: USB hub found [ 4.904675] hub 4-0:1.0: 2 ports detected [ 4.908939] usbcore: registered new interface driver usbserial_generic [ 4.915480] usbserial: USB Serial support registered for generic [ 4.921533] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 5.160269] usb 3-1: new high-speed USB device number 2 using xhci_hcd [ 5.290214] usb 3-1: New USB device found, idVendor=1604, idProduct=10c0 [ 5.296918] usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 5.309227] hub 3-1:1.0: USB hub found [ 5.313211] hub 3-1:1.0: 4 ports detected [ 5.959428] i8042: No controller found [ 5.963194] sched: RT throttling activated [ 5.963203] tsc: Refined TSC clocksource calibration: 1996.249 MHz [ 5.963364] mousedev: PS/2 mouse device common for all mice [ 5.963566] rtc_cmos 00:01: RTC can wake from S4 [ 5.963917] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 5.964019] rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 5.964075] cpuidle: using governor menu [ 5.964317] EFI Variables Facility v0.08 2004-May-17 [ 5.986744] hidraw: raw HID events driver (C) Jiri Kosina [ 5.986837] usbcore: registered new interface driver usbhid [ 5.986838] usbhid: USB HID core driver [ 5.986928] drop_monitor: Initializing network drop monitor service [ 5.987072] TCP: cubic registered [ 5.987075] Initializing XFRM netlink socket [ 5.987283] NET: Registered protocol family 10 [ 5.987746] NET: Registered protocol family 17 [ 5.987749] mpls_gso: MPLS GSO support [ 5.988829] mce: Using 23 MCE banks [ 5.988890] microcode: CPU0: patch_level=0x08001250 [ 5.988900] microcode: CPU1: patch_level=0x08001250 [ 5.988910] microcode: CPU2: patch_level=0x08001250 [ 5.988920] microcode: CPU3: patch_level=0x08001250 [ 5.988934] microcode: CPU4: patch_level=0x08001250 [ 5.988947] microcode: CPU5: patch_level=0x08001250 [ 5.988960] microcode: CPU6: patch_level=0x08001250 [ 5.988972] microcode: CPU7: patch_level=0x08001250 [ 5.988977] microcode: CPU8: patch_level=0x08001250 [ 5.988986] microcode: CPU9: patch_level=0x08001250 [ 5.988995] microcode: CPU10: patch_level=0x08001250 [ 5.989005] microcode: CPU11: patch_level=0x08001250 [ 5.989016] microcode: CPU12: patch_level=0x08001250 [ 5.989027] microcode: CPU13: patch_level=0x08001250 [ 5.989037] microcode: CPU14: patch_level=0x08001250 [ 5.989048] microcode: CPU15: patch_level=0x08001250 [ 5.992295] usb 3-1.1: new high-speed USB device number 3 using xhci_hcd [ 5.992406] microcode: CPU16: patch_level=0x08001250 [ 5.992414] microcode: CPU17: patch_level=0x08001250 [ 5.992427] microcode: CPU18: patch_level=0x08001250 [ 5.992437] microcode: CPU19: patch_level=0x08001250 [ 5.992448] microcode: CPU20: patch_level=0x08001250 [ 5.992459] microcode: CPU21: patch_level=0x08001250 [ 5.992470] microcode: CPU22: patch_level=0x08001250 [ 5.992480] microcode: CPU23: patch_level=0x08001250 [ 5.992490] microcode: CPU24: patch_level=0x08001250 [ 5.992499] microcode: CPU25: patch_level=0x08001250 [ 5.992507] microcode: CPU26: patch_level=0x08001250 [ 5.992515] microcode: CPU27: patch_level=0x08001250 [ 5.992522] microcode: CPU28: patch_level=0x08001250 [ 5.992531] microcode: CPU29: patch_level=0x08001250 [ 5.992539] microcode: CPU30: patch_level=0x08001250 [ 5.992548] microcode: CPU31: patch_level=0x08001250 [ 5.992555] microcode: CPU32: patch_level=0x08001250 [ 5.992564] microcode: CPU33: patch_level=0x08001250 [ 5.992572] microcode: CPU34: patch_level=0x08001250 [ 5.992580] microcode: CPU35: patch_level=0x08001250 [ 5.992591] microcode: CPU36: patch_level=0x08001250 [ 5.992599] microcode: CPU37: patch_level=0x08001250 [ 5.992608] microcode: CPU38: patch_level=0x08001250 [ 5.992618] microcode: CPU39: patch_level=0x08001250 [ 5.992625] microcode: CPU40: patch_level=0x08001250 [ 5.992634] microcode: CPU41: patch_level=0x08001250 [ 5.992642] microcode: CPU42: patch_level=0x08001250 [ 5.992649] microcode: CPU43: patch_level=0x08001250 [ 5.992657] microcode: CPU44: patch_level=0x08001250 [ 5.992666] microcode: CPU45: patch_level=0x08001250 [ 5.992677] microcode: CPU46: patch_level=0x08001250 [ 5.992685] microcode: CPU47: patch_level=0x08001250 [ 5.992736] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 5.992886] PM: Hibernation image not present or could not be loaded. [ 5.992889] Loading compiled-in X.509 certificates [ 5.992916] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 5.992934] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 5.993316] Loaded X.509 cert 'CentOS Linux kernel signing key: 468656045a39b52ff2152c315f6198c3e658f24d' [ 5.993331] registered taskstats version 1 [ 5.995469] Key type trusted registered [ 5.996987] Key type encrypted registered [ 5.997035] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 5.998782] Magic number: 12:205:39 [ 5.998912] memory memory1676: hash matches [ 5.998946] memory memory889: hash matches [ 5.998972] memory memory208: hash matches [ 5.998979] memory memory75: hash matches [ 6.006245] rtc_cmos 00:01: setting system clock to 2020-03-18 18:02:59 UTC (1584554579) [ 6.394089] Switched to clocksource tsc [ 6.399018] Freeing unused kernel memory: 1876k freed [ 6.404315] Write protecting the kernel read-only data: 12288k [ 6.405236] usb 3-1.1: New USB device found, idVendor=1604, idProduct=10c0 [ 6.405238] usb 3-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 6.425740] Freeing unused kernel memory: 504k freed [ 6.429255] hub 3-1.1:1.0: USB hub found [ 6.429610] hub 3-1.1:1.0: 4 ports detected [ 6.440223] Freeing unused kernel memory: 596k freed [ 6.492515] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 6.493292] usb 3-1.4: new high-speed USB device number 4 using xhci_hcd [ 6.509289] usb 1-1: new high-speed USB device number 2 using xhci_hcd [ 6.524849] systemd[1]: Detected architecture x86-64. [ 6.529922] systemd[1]: Running in initial RAM disk. [ 6.543353] systemd[1]: Set hostname to . [ 6.567241] usb 3-1.4: New USB device found, idVendor=1604, idProduct=10c0 [ 6.574119] usb 3-1.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 6.578572] systemd[1]: Reached target Swap. [ 6.589262] hub 3-1.4:1.0: USB hub found [ 6.593330] systemd[1]: Reached target Timers. [ 6.593489] hub 3-1.4:1.0: 4 ports detected [ 6.606535] systemd[1]: Created slice Root Slice. [ 6.617400] systemd[1]: Listening on Journal Socket. [ 6.628410] systemd[1]: Created slice System Slice. [ 6.639852] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 6.651703] usb 1-1: New USB device found, idVendor=0424, idProduct=2744 [ 6.658774] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 6.667293] usb 1-1: Product: USB2734 [ 6.672344] usb 1-1: Manufacturer: Microchip Tech [ 6.680819] systemd[1]: Starting Apply Kernel Variables... [ 6.691231] hub 1-1:1.0: USB hub found [ 6.695203] hub 1-1:1.0: 4 ports detected [ 6.695428] systemd[1]: Starting Setup Virtual Console... [ 6.709715] systemd[1]: Starting Journal Service... [ 6.732371] systemd[1]: Listening on udev Kernel Socket. [ 6.743353] systemd[1]: Reached target Slices. [ 6.752849] systemd[1]: Starting dracut cmdline hook... [ 6.762340] usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd [ 6.762354] systemd[1]: Reached target Local File Systems. [ 6.780394] systemd[1]: Listening on udev Control Socket. [ 6.789455] usb 2-1: New USB device found, idVendor=0424, idProduct=5744 [ 6.796786] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=0 [ 6.804354] usb 2-1: Product: USB5734 [ 6.808028] usb 2-1: Manufacturer: Microchip Tech [ 6.812787] systemd[1]: Reached target Sockets. [ 6.819230] hub 2-1:1.0: USB hub found [ 6.824204] hub 2-1:1.0: 4 ports detected [ 6.829279] systemd[1]: Started Journal Service. [ 6.834228] usb: port power management may be unreliable [ 6.974940] pps_core: LinuxPPS API ver. 1 registered [ 6.979910] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 6.992337] PTP clock support registered [ 6.997027] mlx_compat: loading out-of-tree module taints kernel. [ 6.997060] megasas: 07.705.02.00-rh1 [ 6.997333] megaraid_sas 0000:c1:00.0: FW now in Ready state [ 6.997336] megaraid_sas 0000:c1:00.0: 64 bit DMA mask and 32 bit consistent mask [ 6.997608] megaraid_sas 0000:c1:00.0: irq 68 for MSI/MSI-X [ 6.997632] megaraid_sas 0000:c1:00.0: irq 69 for MSI/MSI-X [ 6.997659] megaraid_sas 0000:c1:00.0: irq 70 for MSI/MSI-X [ 6.997682] megaraid_sas 0000:c1:00.0: irq 71 for MSI/MSI-X [ 6.997705] megaraid_sas 0000:c1:00.0: irq 72 for MSI/MSI-X [ 6.997728] megaraid_sas 0000:c1:00.0: irq 73 for MSI/MSI-X [ 6.997750] megaraid_sas 0000:c1:00.0: irq 74 for MSI/MSI-X [ 6.997773] megaraid_sas 0000:c1:00.0: irq 75 for MSI/MSI-X [ 6.997795] megaraid_sas 0000:c1:00.0: irq 76 for MSI/MSI-X [ 6.997817] megaraid_sas 0000:c1:00.0: irq 77 for MSI/MSI-X [ 6.997838] megaraid_sas 0000:c1:00.0: irq 78 for MSI/MSI-X [ 6.997860] megaraid_sas 0000:c1:00.0: irq 79 for MSI/MSI-X [ 6.997887] megaraid_sas 0000:c1:00.0: irq 80 for MSI/MSI-X [ 6.997908] megaraid_sas 0000:c1:00.0: irq 81 for MSI/MSI-X [ 6.997932] megaraid_sas 0000:c1:00.0: irq 82 for MSI/MSI-X [ 6.997954] megaraid_sas 0000:c1:00.0: irq 83 for MSI/MSI-X [ 6.997976] megaraid_sas 0000:c1:00.0: irq 84 for MSI/MSI-X [ 6.997998] megaraid_sas 0000:c1:00.0: irq 85 for MSI/MSI-X [ 6.998020] megaraid_sas 0000:c1:00.0: irq 86 for MSI/MSI-X [ 6.998043] megaraid_sas 0000:c1:00.0: irq 87 for MSI/MSI-X [ 6.998067] megaraid_sas 0000:c1:00.0: irq 88 for MSI/MSI-X [ 6.998089] megaraid_sas 0000:c1:00.0: irq 89 for MSI/MSI-X [ 6.998111] megaraid_sas 0000:c1:00.0: irq 90 for MSI/MSI-X [ 6.998133] megaraid_sas 0000:c1:00.0: irq 91 for MSI/MSI-X [ 6.998160] megaraid_sas 0000:c1:00.0: irq 92 for MSI/MSI-X [ 6.998184] megaraid_sas 0000:c1:00.0: irq 93 for MSI/MSI-X [ 6.998207] megaraid_sas 0000:c1:00.0: irq 94 for MSI/MSI-X [ 6.998230] megaraid_sas 0000:c1:00.0: irq 95 for MSI/MSI-X [ 6.998253] megaraid_sas 0000:c1:00.0: irq 96 for MSI/MSI-X [ 6.998274] megaraid_sas 0000:c1:00.0: irq 97 for MSI/MSI-X [ 6.998302] megaraid_sas 0000:c1:00.0: irq 98 for MSI/MSI-X [ 6.998324] megaraid_sas 0000:c1:00.0: irq 99 for MSI/MSI-X [ 6.998346] megaraid_sas 0000:c1:00.0: irq 100 for MSI/MSI-X [ 6.998368] megaraid_sas 0000:c1:00.0: irq 101 for MSI/MSI-X [ 6.998390] megaraid_sas 0000:c1:00.0: irq 102 for MSI/MSI-X [ 6.998411] megaraid_sas 0000:c1:00.0: irq 103 for MSI/MSI-X [ 6.998435] megaraid_sas 0000:c1:00.0: irq 104 for MSI/MSI-X [ 6.998456] megaraid_sas 0000:c1:00.0: irq 105 for MSI/MSI-X [ 6.998477] megaraid_sas 0000:c1:00.0: irq 106 for MSI/MSI-X [ 6.998498] megaraid_sas 0000:c1:00.0: irq 107 for MSI/MSI-X [ 6.998519] megaraid_sas 0000:c1:00.0: irq 108 for MSI/MSI-X [ 6.998539] megaraid_sas 0000:c1:00.0: irq 109 for MSI/MSI-X [ 6.998560] megaraid_sas 0000:c1:00.0: irq 110 for MSI/MSI-X [ 6.998581] megaraid_sas 0000:c1:00.0: irq 111 for MSI/MSI-X [ 6.998602] megaraid_sas 0000:c1:00.0: irq 112 for MSI/MSI-X [ 6.998625] megaraid_sas 0000:c1:00.0: irq 113 for MSI/MSI-X [ 6.998646] megaraid_sas 0000:c1:00.0: irq 114 for MSI/MSI-X [ 6.998666] megaraid_sas 0000:c1:00.0: irq 115 for MSI/MSI-X [ 6.998775] megaraid_sas 0000:c1:00.0: firmware supports msix : (96) [ 6.998776] megaraid_sas 0000:c1:00.0: current msix/online cpus : (48/48) [ 6.998777] megaraid_sas 0000:c1:00.0: RDPQ mode : (disabled) [ 6.998780] megaraid_sas 0000:c1:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 237 [ 6.999038] megaraid_sas 0000:c1:00.0: Configured max firmware commands: 927 [ 7.001042] megaraid_sas 0000:c1:00.0: FW supports sync cache : No [ 7.015085] mpt3sas: loading out-of-tree module taints kernel. [ 7.059906] mpt3sas: module verification failed: signature and/or required key missing - tainting kernel [ 7.086102] tg3.c:v3.137 (May 11, 2014) [ 7.088146] libata version 3.00 loaded. [ 7.091075] Compat-mlnx-ofed backport release: 1c4bf42 [ 7.091358] mpt3sas version 31.00.00.00 loaded [ 7.096092] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (263564432 kB) [ 7.103358] tg3 0000:81:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 4c:d9:8f:48:5a:bf [ 7.103361] tg3 0000:81:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 7.103364] tg3 0000:81:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 7.103366] tg3 0000:81:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] [ 7.119474] Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git 1c4bf42 [ 7.119475] compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git [ 7.123497] tg3 0000:81:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 4c:d9:8f:48:5a:c0 [ 7.123501] tg3 0000:81:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 7.123503] tg3 0000:81:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 7.123505] tg3 0000:81:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] [ 7.173342] mpt3sas_cm0: IOC Number : 0 [ 7.173442] mpt3sas 0000:84:00.0: irq 119 for MSI/MSI-X [ 7.173463] mpt3sas 0000:84:00.0: irq 120 for MSI/MSI-X [ 7.173484] mpt3sas 0000:84:00.0: irq 121 for MSI/MSI-X [ 7.173504] mpt3sas 0000:84:00.0: irq 122 for MSI/MSI-X [ 7.173525] mpt3sas 0000:84:00.0: irq 123 for MSI/MSI-X [ 7.173552] mpt3sas 0000:84:00.0: irq 124 for MSI/MSI-X [ 7.173584] mpt3sas 0000:84:00.0: irq 125 for MSI/MSI-X [ 7.173603] mpt3sas 0000:84:00.0: irq 126 for MSI/MSI-X [ 7.173623] mpt3sas 0000:84:00.0: irq 127 for MSI/MSI-X [ 7.173651] mpt3sas 0000:84:00.0: irq 128 for MSI/MSI-X [ 7.173673] mpt3sas 0000:84:00.0: irq 129 for MSI/MSI-X [ 7.173693] mpt3sas 0000:84:00.0: irq 130 for MSI/MSI-X [ 7.173713] mpt3sas 0000:84:00.0: irq 131 for MSI/MSI-X [ 7.173736] mpt3sas 0000:84:00.0: irq 132 for MSI/MSI-X [ 7.173757] mpt3sas 0000:84:00.0: irq 133 for MSI/MSI-X [ 7.173776] mpt3sas 0000:84:00.0: irq 134 for MSI/MSI-X [ 7.173797] mpt3sas 0000:84:00.0: irq 135 for MSI/MSI-X [ 7.173817] mpt3sas 0000:84:00.0: irq 136 for MSI/MSI-X [ 7.173837] mpt3sas 0000:84:00.0: irq 137 for MSI/MSI-X [ 7.173856] mpt3sas 0000:84:00.0: irq 138 for MSI/MSI-X [ 7.173876] mpt3sas 0000:84:00.0: irq 139 for MSI/MSI-X [ 7.173897] mpt3sas 0000:84:00.0: irq 140 for MSI/MSI-X [ 7.173916] mpt3sas 0000:84:00.0: irq 141 for MSI/MSI-X [ 7.173936] mpt3sas 0000:84:00.0: irq 142 for MSI/MSI-X [ 7.173956] mpt3sas 0000:84:00.0: irq 143 for MSI/MSI-X [ 7.173976] mpt3sas 0000:84:00.0: irq 144 for MSI/MSI-X [ 7.173995] mpt3sas 0000:84:00.0: irq 145 for MSI/MSI-X [ 7.174015] mpt3sas 0000:84:00.0: irq 146 for MSI/MSI-X [ 7.174034] mpt3sas 0000:84:00.0: irq 147 for MSI/MSI-X [ 7.174056] mpt3sas 0000:84:00.0: irq 148 for MSI/MSI-X [ 7.174077] mpt3sas 0000:84:00.0: irq 149 for MSI/MSI-X [ 7.174097] mpt3sas 0000:84:00.0: irq 150 for MSI/MSI-X [ 7.174117] mpt3sas 0000:84:00.0: irq 151 for MSI/MSI-X [ 7.174137] mpt3sas 0000:84:00.0: irq 152 for MSI/MSI-X [ 7.174156] mpt3sas 0000:84:00.0: irq 153 for MSI/MSI-X [ 7.174177] mpt3sas 0000:84:00.0: irq 154 for MSI/MSI-X [ 7.174197] mpt3sas 0000:84:00.0: irq 155 for MSI/MSI-X [ 7.174219] mpt3sas 0000:84:00.0: irq 156 for MSI/MSI-X [ 7.174239] mpt3sas 0000:84:00.0: irq 157 for MSI/MSI-X [ 7.174259] mpt3sas 0000:84:00.0: irq 158 for MSI/MSI-X [ 7.174280] mpt3sas 0000:84:00.0: irq 159 for MSI/MSI-X [ 7.174304] mpt3sas 0000:84:00.0: irq 160 for MSI/MSI-X [ 7.174326] mpt3sas 0000:84:00.0: irq 161 for MSI/MSI-X [ 7.174346] mpt3sas 0000:84:00.0: irq 162 for MSI/MSI-X [ 7.174365] mpt3sas 0000:84:00.0: irq 163 for MSI/MSI-X [ 7.174386] mpt3sas 0000:84:00.0: irq 164 for MSI/MSI-X [ 7.174405] mpt3sas 0000:84:00.0: irq 165 for MSI/MSI-X [ 7.174425] mpt3sas 0000:84:00.0: irq 166 for MSI/MSI-X [ 7.175020] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 119 [ 7.175021] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 120 [ 7.175021] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 121 [ 7.175022] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 122 [ 7.175023] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 123 [ 7.175023] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 124 [ 7.175024] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 125 [ 7.175024] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 126 [ 7.175025] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 127 [ 7.175025] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 128 [ 7.175026] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 129 [ 7.175027] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 130 [ 7.175027] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 131 [ 7.175028] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 132 [ 7.175028] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 133 [ 7.175029] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 134 [ 7.175029] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 135 [ 7.175030] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 136 [ 7.175030] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 137 [ 7.175031] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 138 [ 7.175032] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 139 [ 7.175032] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 140 [ 7.175033] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 141 [ 7.175033] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 142 [ 7.175034] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 143 [ 7.175034] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 144 [ 7.175035] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 145 [ 7.175035] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 146 [ 7.175036] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 147 [ 7.175037] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 148 [ 7.175037] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 149 [ 7.175038] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 150 [ 7.175038] mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 151 [ 7.175039] mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 152 [ 7.175039] mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 153 [ 7.175040] mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 154 [ 7.175040] mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 155 [ 7.175041] mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 156 [ 7.175042] mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 157 [ 7.175042] mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 158 [ 7.175043] mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 159 [ 7.175043] mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 160 [ 7.175044] mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 161 [ 7.175044] mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 162 [ 7.175045] mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 163 [ 7.175045] mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 164 [ 7.175046] mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 165 [ 7.175047] mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 166 [ 7.175049] mpt3sas_cm0: iomem(0x00000000ac000000), mapped(0xffffb8d19a000000), size(1048576) [ 7.175050] mpt3sas_cm0: ioport(0x0000000000008000), size(256) [ 7.251313] mpt3sas_cm0: IOC Number : 0 [ 7.251320] mpt3sas_cm0: sending message unit reset !! [ 7.253316] mpt3sas_cm0: message unit reset: SUCCESS [ 7.357311] megaraid_sas 0000:c1:00.0: Init cmd return status SUCCESS for SCSI host 0 [ 7.378313] megaraid_sas 0000:c1:00.0: firmware type : Legacy(64 VD) firmware [ 7.378314] megaraid_sas 0000:c1:00.0: controller type : iMR(0MB) [ 7.378315] megaraid_sas 0000:c1:00.0: Online Controller Reset(OCR) : Enabled [ 7.378316] megaraid_sas 0000:c1:00.0: Secure JBOD support : No [ 7.378317] megaraid_sas 0000:c1:00.0: NVMe passthru support : No [ 7.399822] megaraid_sas 0000:c1:00.0: INIT adapter done [ 7.399824] megaraid_sas 0000:c1:00.0: Jbod map is not supported megasas_setup_jbod_map 5146 [ 7.421595] mpt3sas_cm0: Allocated physical memory: size(38831 kB) [ 7.421597] mpt3sas_cm0: Current Controller Queue Depth(7564), Max Controller Queue Depth(7680) [ 7.421598] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 7.422726] ahci 0000:86:00.2: version 3.0 [ 7.423119] ahci 0000:86:00.2: irq 168 for MSI/MSI-X [ 7.423123] ahci 0000:86:00.2: irq 169 for MSI/MSI-X [ 7.423126] ahci 0000:86:00.2: irq 170 for MSI/MSI-X [ 7.423129] ahci 0000:86:00.2: irq 171 for MSI/MSI-X [ 7.423133] ahci 0000:86:00.2: irq 172 for MSI/MSI-X [ 7.423136] ahci 0000:86:00.2: irq 173 for MSI/MSI-X [ 7.423139] ahci 0000:86:00.2: irq 174 for MSI/MSI-X [ 7.423143] ahci 0000:86:00.2: irq 175 for MSI/MSI-X [ 7.423146] ahci 0000:86:00.2: irq 176 for MSI/MSI-X [ 7.423149] ahci 0000:86:00.2: irq 177 for MSI/MSI-X [ 7.423152] ahci 0000:86:00.2: irq 178 for MSI/MSI-X [ 7.423155] ahci 0000:86:00.2: irq 179 for MSI/MSI-X [ 7.423158] ahci 0000:86:00.2: irq 180 for MSI/MSI-X [ 7.423161] ahci 0000:86:00.2: irq 181 for MSI/MSI-X [ 7.423164] ahci 0000:86:00.2: irq 182 for MSI/MSI-X [ 7.423167] ahci 0000:86:00.2: irq 183 for MSI/MSI-X [ 7.423208] ahci 0000:86:00.2: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode [ 7.423211] ahci 0000:86:00.2: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part [ 7.423466] scsi host2: ahci [ 7.423527] ata1: SATA max UDMA/133 abar m4096@0xc0a02000 port 0xc0a02100 irq 168 [ 7.426048] megaraid_sas 0000:c1:00.0: pci id : (0x1000)/(0x005f)/(0x1028)/(0x1f4b) [ 7.426050] megaraid_sas 0000:c1:00.0: unevenspan support : yes [ 7.426051] megaraid_sas 0000:c1:00.0: firmware crash dump : no [ 7.426053] megaraid_sas 0000:c1:00.0: jbod sync map : no [ 7.426059] scsi host0: Avago SAS based MegaRAID driver [ 7.445644] scsi 0:2:0:0: Direct-Access DELL PERC H330 Mini 4.30 PQ: 0 ANSI: 5 [ 7.564913] mpt3sas_cm0: FW Package Version(12.00.00.00) [ 7.565166] mpt3sas_cm0: SAS3616: FWVersion(12.00.00.00), ChipRevision(0x02), BiosVersion(00.00.00.00) [ 7.565171] mpt3sas_cm0: Protocol=(Initiator,Target,NVMe), Capabilities=(TLR,EEDP,Diag Trace Buffer,Task Set Full,NCQ) [ 7.565236] mpt3sas 0000:84:00.0: Enabled Extended Tags as Controller Supports [ 7.565251] mpt3sas_cm0: : host protection capabilities enabled DIF1 DIF2 DIF3 [ 7.565262] scsi host1: Fusion MPT SAS Host [ 7.565514] mpt3sas_cm0: sending port enable !! [ 7.565800] mpt3sas_cm0: hba_port entry: ffff8b9ba2b88100, port: 255 is added to hba_port list [ 7.568239] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00deb48a0), phys(21) [ 7.568947] mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa1fa24), phy(0) [ 7.568950] mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) [ 7.569837] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) [ 7.570373] scsi 1:0:0:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 7.570457] scsi 1:0:0:0: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa24), phy(0), device_name(0x500a0984dfa1fa24) [ 7.570458] scsi 1:0:0:0: enclosure logical id(0x300605b00d1148a0), slot(13) [ 7.570459] scsi 1:0:0:0: enclosure level(0x0000), connector name( C3 ) [ 7.570461] scsi 1:0:0:0: serial_number(021825001369 ) [ 7.570463] scsi 1:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 7.647168] mlx5_core 0000:01:00.0: firmware version: 20.26.1040 [ 7.647196] mlx5_core 0000:01:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16 GT/s x16 link) [ 7.728329] ata1: SATA link down (SStatus 0 SControl 300) [ 7.900097] mlx5_core 0000:01:00.0: irq 185 for MSI/MSI-X [ 7.900121] mlx5_core 0000:01:00.0: irq 186 for MSI/MSI-X [ 7.900144] mlx5_core 0000:01:00.0: irq 187 for MSI/MSI-X [ 7.900165] mlx5_core 0000:01:00.0: irq 188 for MSI/MSI-X [ 7.900185] mlx5_core 0000:01:00.0: irq 189 for MSI/MSI-X [ 7.900205] mlx5_core 0000:01:00.0: irq 190 for MSI/MSI-X [ 7.900225] mlx5_core 0000:01:00.0: irq 191 for MSI/MSI-X [ 7.900244] mlx5_core 0000:01:00.0: irq 192 for MSI/MSI-X [ 7.900263] mlx5_core 0000:01:00.0: irq 193 for MSI/MSI-X [ 7.900281] mlx5_core 0000:01:00.0: irq 194 for MSI/MSI-X [ 7.900300] mlx5_core 0000:01:00.0: irq 195 for MSI/MSI-X [ 7.900326] mlx5_core 0000:01:00.0: irq 196 for MSI/MSI-X [ 7.900346] mlx5_core 0000:01:00.0: irq 197 for MSI/MSI-X [ 7.900364] mlx5_core 0000:01:00.0: irq 198 for MSI/MSI-X [ 7.900383] mlx5_core 0000:01:00.0: irq 199 for MSI/MSI-X [ 7.900402] mlx5_core 0000:01:00.0: irq 200 for MSI/MSI-X [ 7.900420] mlx5_core 0000:01:00.0: irq 201 for MSI/MSI-X [ 7.900443] mlx5_core 0000:01:00.0: irq 202 for MSI/MSI-X [ 7.900463] mlx5_core 0000:01:00.0: irq 203 for MSI/MSI-X [ 7.900482] mlx5_core 0000:01:00.0: irq 204 for MSI/MSI-X [ 7.900502] mlx5_core 0000:01:00.0: irq 205 for MSI/MSI-X [ 7.900520] mlx5_core 0000:01:00.0: irq 206 for MSI/MSI-X [ 7.900540] mlx5_core 0000:01:00.0: irq 207 for MSI/MSI-X [ 7.900559] mlx5_core 0000:01:00.0: irq 208 for MSI/MSI-X [ 7.900578] mlx5_core 0000:01:00.0: irq 209 for MSI/MSI-X [ 7.900596] mlx5_core 0000:01:00.0: irq 210 for MSI/MSI-X [ 7.900614] mlx5_core 0000:01:00.0: irq 211 for MSI/MSI-X [ 7.900634] mlx5_core 0000:01:00.0: irq 212 for MSI/MSI-X [ 7.900653] mlx5_core 0000:01:00.0: irq 213 for MSI/MSI-X [ 7.900673] mlx5_core 0000:01:00.0: irq 214 for MSI/MSI-X [ 7.900692] mlx5_core 0000:01:00.0: irq 215 for MSI/MSI-X [ 7.900711] mlx5_core 0000:01:00.0: irq 216 for MSI/MSI-X [ 7.900730] mlx5_core 0000:01:00.0: irq 217 for MSI/MSI-X [ 7.900749] mlx5_core 0000:01:00.0: irq 218 for MSI/MSI-X [ 7.900768] mlx5_core 0000:01:00.0: irq 219 for MSI/MSI-X [ 7.900786] mlx5_core 0000:01:00.0: irq 220 for MSI/MSI-X [ 7.900805] mlx5_core 0000:01:00.0: irq 221 for MSI/MSI-X [ 7.900825] mlx5_core 0000:01:00.0: irq 222 for MSI/MSI-X [ 7.900844] mlx5_core 0000:01:00.0: irq 223 for MSI/MSI-X [ 7.900864] mlx5_core 0000:01:00.0: irq 224 for MSI/MSI-X [ 7.900883] mlx5_core 0000:01:00.0: irq 225 for MSI/MSI-X [ 7.900904] mlx5_core 0000:01:00.0: irq 226 for MSI/MSI-X [ 7.900923] mlx5_core 0000:01:00.0: irq 227 for MSI/MSI-X [ 7.900943] mlx5_core 0000:01:00.0: irq 228 for MSI/MSI-X [ 7.900962] mlx5_core 0000:01:00.0: irq 229 for MSI/MSI-X [ 7.900981] mlx5_core 0000:01:00.0: irq 230 for MSI/MSI-X [ 7.901001] mlx5_core 0000:01:00.0: irq 231 for MSI/MSI-X [ 7.901020] mlx5_core 0000:01:00.0: irq 232 for MSI/MSI-X [ 7.901038] mlx5_core 0000:01:00.0: irq 233 for MSI/MSI-X [ 7.902050] mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged [ 7.909551] mlx5_core 0000:01:00.0: mlx5_pcie_event:303:(pid 6): PCIe slot advertised sufficient power (27W). [ 7.912431] scsi 1:0:0:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 7.912514] scsi 1:0:0:1: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa24), phy(0), device_name(0x500a0984dfa1fa24) [ 7.912515] scsi 1:0:0:1: enclosure logical id(0x300605b00d1148a0), slot(13) [ 7.912517] scsi 1:0:0:1: enclosure level(0x0000), connector name( C3 ) [ 7.912518] scsi 1:0:0:1: serial_number(021825001369 ) [ 7.912521] scsi 1:0:0:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 7.912779] scsi 1:0:0:1: Mode parameters changed [ 7.980187] mlx5_core 0000:01:00.0: mlx5_fw_tracer_start:776:(pid 301): FWTracer: Ownership granted and active [ 7.993561] scsi 1:0:0:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 8.001575] sd 0:2:0:0: [sda] 467664896 512-byte logical blocks: (239 GB/223 GiB) [ 8.001728] sd 0:2:0:0: [sda] Write Protect is off [ 8.001730] sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08 [ 8.001753] sd 0:2:0:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA [ 8.003896] sda: sda1 sda2 sda3 [ 8.004285] sd 0:2:0:0: [sda] Attached SCSI disk [ 8.030666] scsi 1:0:0:31: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa24), phy(0), device_name(0x500a0984dfa1fa24) [ 8.041265] scsi 1:0:0:31: enclosure logical id(0x300605b00d1148a0), slot(13) [ 8.048483] scsi 1:0:0:31: enclosure level(0x0000), connector name( C3 ) [ 8.055294] scsi 1:0:0:31: serial_number(021825001369 ) [ 8.060786] scsi 1:0:0:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.079429] mlx5_ib: Mellanox Connect-IB Infiniband driver v4.7-1.0.0 [ 8.090597] mpt3sas_cm0: detecting: handle(0x0019), sas_address(0x500a0984dfa20c10), phy(4) [ 8.098950] mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(0) [ 8.105711] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0019), lun(0) [ 8.112314] scsi 1:0:1:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.120499] scsi 1:0:1:0: SSP: handle(0x0019), sas_addr(0x500a0984dfa20c10), phy(4), device_name(0x500a0984dfa20c10) [ 8.131011] scsi 1:0:1:0: enclosure logical id(0x300605b00d1148a0), slot(9) [ 8.138055] scsi 1:0:1:0: enclosure level(0x0000), connector name( C2 ) [ 8.144775] scsi 1:0:1:0: serial_number(021825001558 ) [ 8.150176] scsi 1:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.174171] scsi 1:0:1:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.182342] scsi 1:0:1:1: SSP: handle(0x0019), sas_addr(0x500a0984dfa20c10), phy(4), device_name(0x500a0984dfa20c10) [ 8.192858] scsi 1:0:1:1: enclosure logical id(0x300605b00d1148a0), slot(9) [ 8.199903] scsi 1:0:1:1: enclosure level(0x0000), connector name( C2 ) [ 8.206607] scsi 1:0:1:1: serial_number(021825001558 ) [ 8.212014] scsi 1:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.234553] scsi 1:0:1:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 8.243477] scsi 1:0:1:31: SSP: handle(0x0019), sas_addr(0x500a0984dfa20c10), phy(4), device_name(0x500a0984dfa20c10) [ 8.254082] scsi 1:0:1:31: enclosure logical id(0x300605b00d1148a0), slot(9) [ 8.261213] scsi 1:0:1:31: enclosure level(0x0000), connector name( C2 ) [ 8.268009] scsi 1:0:1:31: serial_number(021825001558 ) [ 8.273496] scsi 1:0:1:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.288973] random: crng init done [ 8.299598] mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984db2fa924), phy(8) [ 8.307950] mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) [ 8.314851] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) [ 8.321441] scsi 1:0:2:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.329618] scsi 1:0:2:0: SSP: handle(0x0017), sas_addr(0x500a0984db2fa924), phy(8), device_name(0x500a0984db2fa924) [ 8.340133] scsi 1:0:2:0: enclosure logical id(0x300605b00d1148a0), slot(5) [ 8.347181] scsi 1:0:2:0: enclosure level(0x0000), connector name( C1 ) [ 8.353898] scsi 1:0:2:0: serial_number(021815000354 ) [ 8.359299] scsi 1:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.382335] scsi 1:0:2:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.390497] scsi 1:0:2:1: SSP: handle(0x0017), sas_addr(0x500a0984db2fa924), phy(8), device_name(0x500a0984db2fa924) [ 8.401011] scsi 1:0:2:1: enclosure logical id(0x300605b00d1148a0), slot(5) [ 8.408056] scsi 1:0:2:1: enclosure level(0x0000), connector name( C1 ) [ 8.414774] scsi 1:0:2:1: serial_number(021815000354 ) [ 8.420175] scsi 1:0:2:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.429374] scsi 1:0:2:1: Mode parameters changed [ 8.445543] scsi 1:0:2:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.453734] scsi 1:0:2:2: SSP: handle(0x0017), sas_addr(0x500a0984db2fa924), phy(8), device_name(0x500a0984db2fa924) [ 8.464243] scsi 1:0:2:2: enclosure logical id(0x300605b00d1148a0), slot(5) [ 8.471291] scsi 1:0:2:2: enclosure level(0x0000), connector name( C1 ) [ 8.478008] scsi 1:0:2:2: serial_number(021815000354 ) [ 8.483414] scsi 1:0:2:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.492628] scsi 1:0:2:2: Mode parameters changed [ 8.508549] scsi 1:0:2:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 8.516817] scsi 1:0:2:31: SSP: handle(0x0017), sas_addr(0x500a0984db2fa924), phy(8), device_name(0x500a0984db2fa924) [ 8.527417] scsi 1:0:2:31: enclosure logical id(0x300605b00d1148a0), slot(5) [ 8.534550] scsi 1:0:2:31: enclosure level(0x0000), connector name( C1 ) [ 8.541356] scsi 1:0:2:31: serial_number(021815000354 ) [ 8.546842] scsi 1:0:2:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.569603] mpt3sas_cm0: detecting: handle(0x001a), sas_address(0x500a0984da0f9b10), phy(12) [ 8.578047] mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(0) [ 8.584877] mpt3sas_cm0: TEST_UNIT_READY: handle(0x001a), lun(0) [ 8.591453] scsi 1:0:3:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.599637] scsi 1:0:3:0: SSP: handle(0x001a), sas_addr(0x500a0984da0f9b10), phy(12), device_name(0x500a0984da0f9b10) [ 8.610237] scsi 1:0:3:0: enclosure logical id(0x300605b00d1148a0), slot(1) [ 8.617284] scsi 1:0:3:0: enclosure level(0x0000), connector name( C0 ) [ 8.624003] scsi 1:0:3:0: serial_number(021812047179 ) [ 8.629402] scsi 1:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.650162] scsi 1:0:3:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.658332] scsi 1:0:3:1: SSP: handle(0x001a), sas_addr(0x500a0984da0f9b10), phy(12), device_name(0x500a0984da0f9b10) [ 8.668930] scsi 1:0:3:1: enclosure logical id(0x300605b00d1148a0), slot(1) [ 8.675975] scsi 1:0:3:1: enclosure level(0x0000), connector name( C0 ) [ 8.682693] scsi 1:0:3:1: serial_number(021812047179 ) [ 8.688097] scsi 1:0:3:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.710551] scsi 1:0:3:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.718711] scsi 1:0:3:2: SSP: handle(0x001a), sas_addr(0x500a0984da0f9b10), phy(12), device_name(0x500a0984da0f9b10) [ 8.729312] scsi 1:0:3:2: enclosure logical id(0x300605b00d1148a0), slot(1) [ 8.736358] scsi 1:0:3:2: enclosure level(0x0000), connector name( C0 ) [ 8.743077] scsi 1:0:3:2: serial_number(021812047179 ) [ 8.748478] scsi 1:0:3:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.768549] scsi 1:0:3:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 8.776799] scsi 1:0:3:31: SSP: handle(0x001a), sas_addr(0x500a0984da0f9b10), phy(12), device_name(0x500a0984da0f9b10) [ 8.787485] scsi 1:0:3:31: enclosure logical id(0x300605b00d1148a0), slot(1) [ 8.794618] scsi 1:0:3:31: enclosure level(0x0000), connector name( C0 ) [ 8.801422] scsi 1:0:3:31: serial_number(021812047179 ) [ 8.806911] scsi 1:0:3:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.827194] mpt3sas_cm0: detecting: handle(0x0011), sas_address(0x300705b00deb48a0), phy(16) [ 8.835634] mpt3sas_cm0: REPORT_LUNS: handle(0x0011), retries(0) [ 8.841660] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0011), lun(0) [ 8.848041] scsi 1:0:4:0: Enclosure LSI VirtualSES 03 PQ: 0 ANSI: 7 [ 8.856170] scsi 1:0:4:0: set ignore_delay_remove for handle(0x0011) [ 8.862524] scsi 1:0:4:0: SES: handle(0x0011), sas_addr(0x300705b00deb48a0), phy(16), device_name(0x300705b00deb48a0) [ 8.873124] scsi 1:0:4:0: enclosure logical id(0x300605b00d1148a0), slot(16) [ 8.880254] scsi 1:0:4:0: enclosure level(0x0000), connector name( C3 ) [ 8.886975] scsi 1:0:4:0: serial_number(300605B00D1148A0) [ 8.892376] scsi 1:0:4:0: qdepth(1), tagged(0), simple(0), ordered(0), scsi_level(8), cmd_que(0) [ 8.901181] mpt3sas_cm0: log_info(0x31200206): originator(PL), code(0x20), sub_code(0x0206) [ 8.924343] mpt3sas_cm0: port enable: SUCCESS [ 8.929194] scsi 1:0:0:0: rdac: LUN 0 (IOSHIP) (owned) [ 8.934542] sd 1:0:0:0: [sdb] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 8.942715] scsi 1:0:0:1: rdac: LUN 1 (IOSHIP) (unowned) [ 8.948134] sd 1:0:0:0: [sdb] Write Protect is off [ 8.948248] sd 1:0:0:1: [sdc] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 8.948641] sd 1:0:0:1: [sdc] Write Protect is off [ 8.948642] sd 1:0:0:1: [sdc] Mode Sense: 83 00 10 08 [ 8.948673] scsi 1:0:1:0: rdac: LUN 0 (IOSHIP) (unowned) [ 8.948785] sd 1:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.948896] sd 1:0:1:0: [sdd] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 8.949292] scsi 1:0:1:1: rdac: LUN 1 (IOSHIP) (owned) [ 8.949460] sd 1:0:1:0: [sdd] Write Protect is off [ 8.949462] sd 1:0:1:0: [sdd] Mode Sense: 83 00 10 08 [ 8.949580] sd 1:0:1:1: [sde] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 8.949642] sd 1:0:1:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.949891] scsi 1:0:2:0: rdac: LUN 0 (IOSHIP) (owned) [ 8.950133] sd 1:0:2:0: [sdf] 926167040 512-byte logical blocks: (474 GB/441 GiB) [ 8.950135] sd 1:0:2:0: [sdf] 4096-byte physical blocks [ 8.950374] sd 1:0:1:1: [sde] Write Protect is off [ 8.950375] sd 1:0:1:1: [sde] Mode Sense: 83 00 10 08 [ 8.950527] scsi 1:0:2:1: rdac: LUN 1 (IOSHIP) (unowned) [ 8.950687] sd 1:0:1:1: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.950695] sd 1:0:2:0: [sdf] Write Protect is off [ 8.950696] sd 1:0:2:0: [sdf] Mode Sense: 83 00 10 08 [ 8.950845] sd 1:0:2:1: [sdg] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 8.950942] sd 1:0:2:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.951301] scsi 1:0:2:2: rdac: LUN 2 (IOSHIP) (owned) [ 8.951628] sd 1:0:2:1: [sdg] Write Protect is off [ 8.951634] sd 1:0:2:1: [sdg] Mode Sense: 83 00 10 08 [ 8.951647] sd 1:0:2:2: [sdh] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 8.951863] sd 1:0:2:1: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.951946] scsi 1:0:3:0: rdac: LUN 0 (IOSHIP) (unowned) [ 8.952175] sd 1:0:3:0: [sdi] 926167040 512-byte logical blocks: (474 GB/441 GiB) [ 8.952177] sd 1:0:3:0: [sdi] 4096-byte physical blocks [ 8.952399] sd 1:0:2:2: [sdh] Write Protect is off [ 8.952401] sd 1:0:2:2: [sdh] Mode Sense: 83 00 10 08 [ 8.952630] sd 1:0:2:2: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.952658] scsi 1:0:3:1: rdac: LUN 1 (IOSHIP) (owned) [ 8.952730] sd 1:0:3:0: [sdi] Write Protect is off [ 8.952731] sd 1:0:3:0: [sdi] Mode Sense: 83 00 10 08 [ 8.952827] sd 1:0:0:1: [sdc] Attached SCSI disk [ 8.952867] sd 1:0:3:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.953176] sd 1:0:3:1: [sdj] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 8.953585] sd 1:0:3:1: [sdj] Write Protect is off [ 8.953587] sd 1:0:3:1: [sdj] Mode Sense: 83 00 10 08 [ 8.953732] sd 1:0:3:1: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.953881] sd 1:0:1:1: [sde] Attached SCSI disk [ 8.954622] sd 1:0:1:0: [sdd] Attached SCSI disk [ 8.954669] scsi 1:0:3:2: rdac: LUN 2 (IOSHIP) (unowned) [ 8.954940] sd 1:0:2:0: [sdf] Attached SCSI disk [ 8.955229] sd 1:0:3:2: [sdk] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 8.956461] sd 1:0:3:2: [sdk] Write Protect is off [ 8.956468] sd 1:0:3:2: [sdk] Mode Sense: 83 00 10 08 [ 8.956826] sd 1:0:3:2: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 8.957367] sd 1:0:2:2: [sdh] Attached SCSI disk [ 8.957959] sd 1:0:2:1: [sdg] Attached SCSI disk [ 8.958013] sd 1:0:3:0: [sdi] Attached SCSI disk [ 8.958128] sd 1:0:3:1: [sdj] Attached SCSI disk [ 8.960470] sd 1:0:3:2: [sdk] Attached SCSI disk [ 9.236339] sd 1:0:0:0: [sdb] Mode Sense: 83 00 10 08 [ 9.236516] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 9.246985] sd 1:0:0:0: [sdb] Attached SCSI disk [ 9.344767] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) [ 9.568029] systemd-journald[359]: Received SIGTERM from PID 1 (systemd). [ 9.605987] SELinux: Disabled at runtime. [ 9.611087] SELinux: Unregistering netfilter hooks [ 9.659367] type=1404 audit(1584554583.159:2): selinux=0 auid=4294967295 ses=4294967295 [ 9.689325] ip_tables: (C) 2000-2006 Netfilter Core Team [ 9.695650] systemd[1]: Inserted module 'ip_tables' [ 9.790914] EXT4-fs (sda2): re-mounted. Opts: (null) [ 9.801889] systemd-journald[4902]: Received request to flush runtime journal from PID 1 [ 9.874497] device-mapper: uevent: version 1.0.3 [ 9.879549] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 9.892292] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 [ 9.900853] piix4_smbus 0000:00:14.0: Using register 0x2e for SMBus port selection [ 9.919030] ACPI Error: No handler for Region [SYSI] (ffff8b8c69e7aa68) [IPMI] (20130517/evregion-162) [ 9.927926] ACPI Error: [ 9.927927] Region IPMI (ID=7) has no handler [ 9.927929] (20130517/exfldio-305) [ 9.927936] ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff8b8c69e775a0), AE_NOT_EXIST (20130517/psparse-536) [ 9.927942] ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff8b8c69e77500), AE_NOT_EXIST (20130517/psparse-536) [ 9.927946] ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753) [ 9.980141] ccp 0000:02:00.2: 3 command queues available [ 9.986023] ccp 0000:02:00.2: irq 235 for MSI/MSI-X [ 9.986047] ccp 0000:02:00.2: irq 236 for MSI/MSI-X [ 9.986091] ccp 0000:02:00.2: Queue 2 can access 4 LSB regions [ 9.993022] ccp 0000:02:00.2: Queue 3 can access 4 LSB regions [ 10.000257] ccp 0000:02:00.2: Queue 4 can access 4 LSB regions [ 10.007477] ccp 0000:02:00.2: Queue 0 gets LSB 4 [ 10.013496] ccp 0000:02:00.2: Queue 1 gets LSB 5 [ 10.013596] input: PC Speaker as /devices/platform/pcspkr/input/input2 [ 10.027433] ccp 0000:02:00.2: Queue 2 gets LSB 6 [ 10.034022] sd 0:2:0:0: Attached scsi generic sg0 type 0 [ 10.034792] ccp 0000:02:00.2: enabled [ 10.035010] ccp 0000:03:00.1: 5 command queues available [ 10.035070] ccp 0000:03:00.1: irq 238 for MSI/MSI-X [ 10.035106] ccp 0000:03:00.1: Queue 0 can access 7 LSB regions [ 10.035108] ccp 0000:03:00.1: Queue 1 can access 7 LSB regions [ 10.035110] ccp 0000:03:00.1: Queue 2 can access 7 LSB regions [ 10.035112] ccp 0000:03:00.1: Queue 3 can access 7 LSB regions [ 10.035114] ccp 0000:03:00.1: Queue 4 can access 7 LSB regions [ 10.035116] ccp 0000:03:00.1: Queue 0 gets LSB 1 [ 10.035117] ccp 0000:03:00.1: Queue 1 gets LSB 2 [ 10.035118] ccp 0000:03:00.1: Queue 2 gets LSB 3 [ 10.035119] ccp 0000:03:00.1: Queue 3 gets LSB 4 [ 10.035120] ccp 0000:03:00.1: Queue 4 gets LSB 5 [ 10.035544] ccp 0000:03:00.1: enabled [ 10.035714] ccp 0000:41:00.2: 3 command queues available [ 10.035758] ccp 0000:41:00.2: irq 240 for MSI/MSI-X [ 10.035779] ccp 0000:41:00.2: irq 241 for MSI/MSI-X [ 10.035825] ccp 0000:41:00.2: Queue 2 can access 4 LSB regions [ 10.035827] ccp 0000:41:00.2: Queue 3 can access 4 LSB regions [ 10.035829] ccp 0000:41:00.2: Queue 4 can access 4 LSB regions [ 10.035831] ccp 0000:41:00.2: Queue 0 gets LSB 4 [ 10.035832] ccp 0000:41:00.2: Queue 1 gets LSB 5 [ 10.035833] ccp 0000:41:00.2: Queue 2 gets LSB 6 [ 10.036200] ccp 0000:41:00.2: enabled [ 10.036328] ccp 0000:42:00.1: 5 command queues available [ 10.036375] ccp 0000:42:00.1: irq 243 for MSI/MSI-X [ 10.036403] ccp 0000:42:00.1: Queue 0 can access 7 LSB regions [ 10.036405] ccp 0000:42:00.1: Queue 1 can access 7 LSB regions [ 10.036407] ccp 0000:42:00.1: Queue 2 can access 7 LSB regions [ 10.036409] ccp 0000:42:00.1: Queue 3 can access 7 LSB regions [ 10.036411] ccp 0000:42:00.1: Queue 4 can access 7 LSB regions [ 10.036412] ccp 0000:42:00.1: Queue 0 gets LSB 1 [ 10.036413] ccp 0000:42:00.1: Queue 1 gets LSB 2 [ 10.036414] ccp 0000:42:00.1: Queue 2 gets LSB 3 [ 10.036415] ccp 0000:42:00.1: Queue 3 gets LSB 4 [ 10.036416] ccp 0000:42:00.1: Queue 4 gets LSB 5 [ 10.037302] ccp 0000:42:00.1: enabled [ 10.037571] ccp 0000:85:00.2: 3 command queues available [ 10.037624] ccp 0000:85:00.2: irq 245 for MSI/MSI-X [ 10.037645] ccp 0000:85:00.2: irq 246 for MSI/MSI-X [ 10.037697] ccp 0000:85:00.2: Queue 2 can access 4 LSB regions [ 10.037699] ccp 0000:85:00.2: Queue 3 can access 4 LSB regions [ 10.037701] ccp 0000:85:00.2: Queue 4 can access 4 LSB regions [ 10.037703] ccp 0000:85:00.2: Queue 0 gets LSB 4 [ 10.037704] ccp 0000:85:00.2: Queue 1 gets LSB 5 [ 10.037705] ccp 0000:85:00.2: Queue 2 gets LSB 6 [ 10.038060] ccp 0000:85:00.2: enabled [ 10.038227] ccp 0000:86:00.1: 5 command queues available [ 10.038279] ccp 0000:86:00.1: irq 248 for MSI/MSI-X [ 10.038308] ccp 0000:86:00.1: Queue 0 can access 7 LSB regions [ 10.038311] ccp 0000:86:00.1: Queue 1 can access 7 LSB regions [ 10.038314] ccp 0000:86:00.1: Queue 2 can access 7 LSB regions [ 10.038316] ccp 0000:86:00.1: Queue 3 can access 7 LSB regions [ 10.038318] ccp 0000:86:00.1: Queue 4 can access 7 LSB regions [ 10.038320] ccp 0000:86:00.1: Queue 0 gets LSB 1 [ 10.038321] ccp 0000:86:00.1: Queue 1 gets LSB 2 [ 10.038323] ccp 0000:86:00.1: Queue 2 gets LSB 3 [ 10.038324] ccp 0000:86:00.1: Queue 3 gets LSB 4 [ 10.038325] ccp 0000:86:00.1: Queue 4 gets LSB 5 [ 10.038743] ccp 0000:86:00.1: enabled [ 10.038944] ccp 0000:c2:00.2: 3 command queues available [ 10.038996] ccp 0000:c2:00.2: irq 250 for MSI/MSI-X [ 10.039023] ccp 0000:c2:00.2: irq 251 for MSI/MSI-X [ 10.039072] ccp 0000:c2:00.2: Queue 2 can access 4 LSB regions [ 10.039074] ccp 0000:c2:00.2: Queue 3 can access 4 LSB regions [ 10.039077] ccp 0000:c2:00.2: Queue 4 can access 4 LSB regions [ 10.039078] ccp 0000:c2:00.2: Queue 0 gets LSB 4 [ 10.039080] ccp 0000:c2:00.2: Queue 1 gets LSB 5 [ 10.039082] ccp 0000:c2:00.2: Queue 2 gets LSB 6 [ 10.039955] ccp 0000:c2:00.2: enabled [ 10.040095] ccp 0000:c3:00.1: 5 command queues available [ 10.040147] ccp 0000:c3:00.1: irq 253 for MSI/MSI-X [ 10.040173] ccp 0000:c3:00.1: Queue 0 can access 7 LSB regions [ 10.040175] ccp 0000:c3:00.1: Queue 1 can access 7 LSB regions [ 10.040177] ccp 0000:c3:00.1: Queue 2 can access 7 LSB regions [ 10.040179] ccp 0000:c3:00.1: Queue 3 can access 7 LSB regions [ 10.040181] ccp 0000:c3:00.1: Queue 4 can access 7 LSB regions [ 10.040183] ccp 0000:c3:00.1: Queue 0 gets LSB 1 [ 10.040184] ccp 0000:c3:00.1: Queue 1 gets LSB 2 [ 10.040185] ccp 0000:c3:00.1: Queue 2 gets LSB 3 [ 10.040187] ccp 0000:c3:00.1: Queue 3 gets LSB 4 [ 10.040188] ccp 0000:c3:00.1: Queue 4 gets LSB 5 [ 10.041140] ccp 0000:c3:00.1: enabled [ 10.112248] cryptd: max_cpu_qlen set to 1000 [ 10.203121] ipmi message handler version 39.2 [ 10.351458] sd 1:0:0:0: Attached scsi generic sg1 type 0 [ 10.351512] sd 1:0:0:1: Attached scsi generic sg2 type 0 [ 10.351558] scsi 1:0:0:31: Attached scsi generic sg3 type 0 [ 10.351605] sd 1:0:1:0: Attached scsi generic sg4 type 0 [ 10.351647] sd 1:0:1:1: Attached scsi generic sg5 type 0 [ 10.351683] scsi 1:0:1:31: Attached scsi generic sg6 type 0 [ 10.351724] sd 1:0:2:0: Attached scsi generic sg7 type 0 [ 10.351762] sd 1:0:2:1: Attached scsi generic sg8 type 0 [ 10.351797] sd 1:0:2:2: Attached scsi generic sg9 type 0 [ 10.351837] scsi 1:0:2:31: Attached scsi generic sg10 type 0 [ 10.351873] sd 1:0:3:0: Attached scsi generic sg11 type 0 [ 10.351910] sd 1:0:3:1: Attached scsi generic sg12 type 0 [ 10.351947] sd 1:0:3:2: Attached scsi generic sg13 type 0 [ 10.351980] scsi 1:0:3:31: Attached scsi generic sg14 type 0 [ 10.352019] scsi 1:0:4:0: Attached scsi generic sg15 type 13 [ 10.568633] ipmi device interface [ 10.568641] AVX2 version of gcm_enc/dec engaged. [ 10.568642] AES CTR mode by8 optimization enabled [ 10.571042] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 10.571114] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 10.657678] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.3) [ 10.667063] sd 1:0:0:0: Embedded Enclosure Device [ 10.677805] sd 1:0:0:1: Embedded Enclosure Device [ 10.696427] scsi 1:0:0:31: Embedded Enclosure Device [ 10.698938] sd 1:0:1:0: Embedded Enclosure Device [ 10.699056] IPMI System Interface driver [ 10.699084] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS [ 10.699087] ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 [ 10.699088] ipmi_si: Adding SMBIOS-specified kcs state machine [ 10.699115] ipmi_si IPI0001:00: ipmi_platform: probing via ACPI [ 10.699143] ipmi_si IPI0001:00: [io 0x0ca8] regsize 1 spacing 4 irq 10 [ 10.699146] ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI [ 10.699147] ipmi_si: Adding ACPI-specified kcs state machine [ 10.699249] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 [ 10.701289] sd 1:0:1:1: Embedded Enclosure Device [ 10.703515] scsi 1:0:1:31: Embedded Enclosure Device [ 10.705868] sd 1:0:2:0: Embedded Enclosure Device [ 10.708199] sd 1:0:2:1: Embedded Enclosure Device [ 10.710415] sd 1:0:2:2: Embedded Enclosure Device [ 10.712548] scsi 1:0:2:31: Embedded Enclosure Device [ 10.714627] sd 1:0:3:0: Embedded Enclosure Device [ 10.716829] sd 1:0:3:1: Embedded Enclosure Device [ 10.718921] sd 1:0:3:2: Embedded Enclosure Device [ 10.721015] scsi 1:0:3:31: Embedded Enclosure Device [ 10.724089] ses 1:0:4:0: Attached Enclosure device [ 10.732436] ipmi_si IPI0001:00: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. [ 10.737572] ipmi_si IPI0001:00: Using irq 10 [ 10.760352] ipmi_si IPI0001:00: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) [ 10.838733] ipmi_si IPI0001:00: IPMI kcs interface initialized [ 10.873363] kvm: Nested Paging enabled [ 10.880079] MCE: In-kernel MCE decoding enabled. [ 10.888016] AMD64 EDAC driver v3.4.0 [ 10.891650] EDAC amd64: DRAM ECC enabled. [ 10.895715] EDAC amd64: F17h detected (node 0). [ 10.900295] EDAC MC: UMC0 chip selects: [ 10.900297] EDAC amd64: MC: 0: 0MB 1: 0MB [ 10.905006] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 10.909713] EDAC amd64: MC: 4: 0MB 5: 0MB [ 10.914421] EDAC amd64: MC: 6: 0MB 7: 0MB [ 10.914426] EDAC MC: UMC1 chip selects: [ 10.914430] EDAC amd64: MC: 0: 0MB 1: 0MB [ 10.914431] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 10.914432] EDAC amd64: MC: 4: 0MB 5: 0MB [ 10.914432] EDAC amd64: MC: 6: 0MB 7: 0MB [ 10.914433] EDAC amd64: using x8 syndromes. [ 10.914433] EDAC amd64: MCT channel count: 2 [ 10.914595] EDAC MC0: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:18.3 [ 10.914601] EDAC amd64: DRAM ECC enabled. [ 10.914602] EDAC amd64: F17h detected (node 1). [ 10.914644] EDAC MC: UMC0 chip selects: [ 10.914644] EDAC amd64: MC: 0: 0MB 1: 0MB [ 10.914645] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 10.914646] EDAC amd64: MC: 4: 0MB 5: 0MB [ 10.914646] EDAC amd64: MC: 6: 0MB 7: 0MB [ 10.914649] EDAC MC: UMC1 chip selects: [ 10.914649] EDAC amd64: MC: 0: 0MB 1: 0MB [ 10.914650] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 10.914651] EDAC amd64: MC: 4: 0MB 5: 0MB [ 10.914651] EDAC amd64: MC: 6: 0MB 7: 0MB [ 10.914651] EDAC amd64: using x8 syndromes. [ 10.914652] EDAC amd64: MCT channel count: 2 [ 10.914813] EDAC MC1: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:19.3 [ 10.914818] EDAC amd64: DRAM ECC enabled. [ 10.914819] EDAC amd64: F17h detected (node 2). [ 10.914861] EDAC MC: UMC0 chip selects: [ 10.914862] EDAC amd64: MC: 0: 0MB 1: 0MB [ 10.914863] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 10.914863] EDAC amd64: MC: 4: 0MB 5: 0MB [ 10.914864] EDAC amd64: MC: 6: 0MB 7: 0MB [ 10.914866] EDAC MC: UMC1 chip selects: [ 10.914867] EDAC amd64: MC: 0: 0MB 1: 0MB [ 10.914867] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 10.914868] EDAC amd64: MC: 4: 0MB 5: 0MB [ 10.914869] EDAC amd64: MC: 6: 0MB 7: 0MB [ 10.914869] EDAC amd64: using x8 syndromes. [ 10.914869] EDAC amd64: MCT channel count: 2 [ 10.915751] EDAC MC2: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1a.3 [ 10.915756] EDAC amd64: DRAM ECC enabled. [ 10.915757] EDAC amd64: F17h detected (node 3). [ 10.915797] EDAC MC: UMC0 chip selects: [ 10.915797] EDAC amd64: MC: 0: 0MB 1: 0MB [ 10.915798] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 10.915799] EDAC amd64: MC: 4: 0MB 5: 0MB [ 10.915799] EDAC amd64: MC: 6: 0MB 7: 0MB [ 10.915801] EDAC MC: UMC1 chip selects: [ 10.915802] EDAC amd64: MC: 0: 0MB 1: 0MB [ 10.915803] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 10.915803] EDAC amd64: MC: 4: 0MB 5: 0MB [ 10.915804] EDAC amd64: MC: 6: 0MB 7: 0MB [ 10.915804] EDAC amd64: using x8 syndromes. [ 10.915804] EDAC amd64: MCT channel count: 2 [ 10.916677] EDAC MC3: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1b.3 [ 10.916751] EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.0' (POLLED) [ 36.964699] device-mapper: multipath round-robin: version 1.2.0 loaded [ 50.330281] Adding 4194300k swap on /dev/sda3. Priority:-2 extents:1 across:4194300k FS [ 50.372464] type=1305 audit(1584554623.871:3): audit_pid=10589 old=0 auid=4294967295 ses=4294967295 res=1 [ 50.392709] RPC: Registered named UNIX socket transport module. [ 50.399426] RPC: Registered udp transport module. [ 50.405517] RPC: Registered tcp transport module. [ 50.411605] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 51.063398] mlx5_core 0000:01:00.0: slow_pci_heuristic:5575:(pid 10887): Max link speed = 100000, PCI BW = 126016 [ 51.073715] mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 51.081997] mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 51.527323] tg3 0000:81:00.0: irq 254 for MSI/MSI-X [ 51.527338] tg3 0000:81:00.0: irq 255 for MSI/MSI-X [ 51.527350] tg3 0000:81:00.0: irq 256 for MSI/MSI-X [ 51.527360] tg3 0000:81:00.0: irq 257 for MSI/MSI-X [ 51.527398] tg3 0000:81:00.0: irq 258 for MSI/MSI-X [ 51.653495] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready [ 55.088424] tg3 0000:81:00.0 em1: Link is up at 1000 Mbps, full duplex [ 55.094958] tg3 0000:81:00.0 em1: Flow control is on for TX and on for RX [ 55.101747] tg3 0000:81:00.0 em1: EEE is enabled [ 55.106385] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready [ 56.006727] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 56.280008] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 60.418448] FS-Cache: Loaded [ 60.448999] FS-Cache: Netfs 'nfs' registered for caching [ 60.458874] Key type dns_resolver registered [ 60.488519] NFS: Registering the id_resolver key type [ 60.493979] Key type id_resolver registered [ 60.499568] Key type id_legacy registered [ 558.674541] LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 [ 558.683053] alg: No test for adler32 (adler32-zlib) [ 559.482924] Lustre: Lustre: Build Version: 2.12.4 [ 559.587271] LNet: 20244:0:(config.c:1627:lnet_inet_enumerate()) lnet: Ignoring interface em2: it's down [ 559.597037] LNet: Using FastReg for registration [ 559.614426] LNet: Added LNI 10.0.10.53@o2ib7 [8/256/0/180] [ 671.303914] LDISKFS-fs (dm-4): file extents enabled, maximum tree depth=5 [ 671.387435] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [ 674.374703] Lustre: fir-MDT0002: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 674.529848] Lustre: fir-MDD0002: changelog on [ 674.539928] Lustre: fir-MDT0002: in recovery but waiting for the first client to connect [ 675.825608] Lustre: fir-MDT0002: Will be in recovery for at least 2:30, or until 1290 clients reconnect [ 676.835302] Lustre: fir-MDT0002: Connection restored to a9a870dd-42ae-4 (at 10.50.16.16@o2ib2) [ 676.843923] Lustre: Skipped 2 previous similar messages [ 677.524584] Lustre: fir-MDT0002: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [ 677.533115] Lustre: Skipped 2 previous similar messages [ 678.529762] Lustre: fir-MDT0002: Connection restored to b9bccb6f-5d4b-4 (at 10.50.5.58@o2ib2) [ 678.538296] Lustre: Skipped 179 previous similar messages [ 680.530334] Lustre: fir-MDT0002: Connection restored to 3f0a0b2b-f13d-4 (at 10.49.31.10@o2ib1) [ 680.538947] Lustre: Skipped 502 previous similar messages [ 692.042781] Lustre: fir-MDT0002: Connection restored to fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) [ 692.052006] Lustre: Skipped 590 previous similar messages [ 702.048975] Lustre: fir-MDT0002: Connection restored to 3946a76d-ba08-4 (at 10.50.9.32@o2ib2) [ 702.057506] Lustre: Skipped 99 previous similar messages [ 714.510343] Lustre: fir-MDT0002: Recovery over after 0:38, of 1290 clients 1290 recovered and 0 were evicted. [ 829.408805] LustreError: 11-0: fir-MDT0003-osp-MDT0002: operation mds_statfs to node 10.0.10.54@o2ib7 failed: rc = -107 [ 829.419595] Lustre: fir-MDT0003-osp-MDT0002: Connection to fir-MDT0003 (at 10.0.10.54@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 848.332477] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.4.14@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.265482] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.9.50@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.664770] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.14.13@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.682141] LustreError: Skipped 2 previous similar messages [ 852.664852] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.7.10@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.682202] LustreError: Skipped 97 previous similar messages [ 856.697451] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.10.42@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.714825] LustreError: Skipped 657 previous similar messages [ 865.186134] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.9.61@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.203416] LustreError: Skipped 138 previous similar messages [ 883.884669] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.0.62@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.901953] LustreError: Skipped 480 previous similar messages [ 948.686989] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.4.14@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.557919] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 961.644260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [ 962.585165] Lustre: fir-MDT0002: Connection restored to 10.0.10.53@o2ib7 (at 0@lo) [ 962.779278] Lustre: fir-MDT0003: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 962.839268] Lustre: fir-MDD0003: changelog on [ 962.856816] Lustre: fir-MDT0003: in recovery but waiting for the first client to connect [ 963.901620] Lustre: fir-MDT0003: Will be in recovery for at least 2:30, or until 1290 clients reconnect [ 1043.598797] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds [ 1043.608921] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 1049.598938] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds [ 1049.609046] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 1053.288638] Lustre: fir-MDT0003: Connection restored to 3946a76d-ba08-4 (at 10.50.9.32@o2ib2) [ 1053.297170] Lustre: Skipped 1381 previous similar messages [ 1055.600095] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds [ 1055.610205] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 1063.171462] Lustre: fir-MDT0003: Recovery over after 1:39, of 1290 clients 1290 recovered and 0 were evicted. [ 3039.909002] Lustre: Failing over fir-MDT0003 [ 3039.941295] Lustre: fir-MDT0003: Not available for connect from 10.0.10.3@o2ib7 (stopping) [ 3039.949568] Lustre: Skipped 1 previous similar message [ 3040.446321] Lustre: fir-MDT0003: Not available for connect from 10.50.17.6@o2ib2 (stopping) [ 3040.454685] Lustre: Skipped 21 previous similar messages [ 3041.382089] LustreError: 11-0: fir-MDT0003-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 [ 3041.391838] Lustre: fir-MDT0003-osp-MDT0002: Connection to fir-MDT0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.471304] Lustre: fir-MDT0003: Not available for connect from 10.50.5.42@o2ib2 (stopping) [ 3041.479667] Lustre: Skipped 47 previous similar messages [ 3043.738297] Lustre: fir-MDT0003: Not available for connect from 10.49.7.5@o2ib1 (stopping) [ 3043.746562] Lustre: Skipped 53 previous similar messages [ 3046.493373] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.10.2@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.510661] LustreError: Skipped 894 previous similar messages [ 3046.805836] Lustre: server umount fir-MDT0003 complete [ 3054.581620] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.7.16@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.598899] LustreError: Skipped 199 previous similar messages [ 3056.797154] Lustre: fir-MDT0002: Connection restored to 10.0.10.54@o2ib7 (at 10.0.10.54@o2ib7) [ 3056.805774] Lustre: Skipped 2 previous similar messages [ 3085.495693] Lustre: fir-MDT0003-osp-MDT0002: Connection restored to 10.0.10.54@o2ib7 (at 10.0.10.54@o2ib7) [ 3213.925013] LustreError: 11-0: fir-MDT0000-osp-MDT0002: operation ldlm_enqueue to node 10.0.10.51@o2ib7 failed: rc = -107 [ 3213.935974] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 3220.730597] Lustre: fir-MDT0000-lwp-MDT0002: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 3258.990320] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.12@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.007600] LustreError: Skipped 940 previous similar messages [ 3302.962993] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.7.2@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.980188] LustreError: Skipped 1386 previous similar messages [ 3366.968938] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.17.1@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.986223] LustreError: Skipped 383 previous similar messages [ 3414.655023] LNet: Service thread pid 20883 was inactive for 200.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 3414.671960] Pid: 20883, comm: mdt00_008 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [ 3414.682140] Call Trace: [ 3414.684604] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [ 3414.691232] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [ 3414.697922] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [ 3414.704595] [] osp_md_object_lock+0x162/0x2d0 [osp] [ 3414.711179] [] lod_object_lock+0xf4/0x780 [lod] [ 3414.717405] [] mdd_object_lock+0x3e/0xe0 [mdd] [ 3414.723531] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [ 3414.730797] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [ 3414.737552] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [ 3414.743778] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [ 3414.750275] [] mdt_reint_rec+0x83/0x210 [mdt] [ 3414.756342] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 3414.762939] [] mdt_reint+0x67/0x140 [mdt] [ 3414.768660] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 3414.775627] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3414.783345] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 3414.789684] [] kthread+0xd1/0xe0 [ 3414.794602] [] ret_from_fork_nospec_begin+0xe/0x21 [ 3414.801091] [] 0xffffffffffffffff [ 3414.806115] LustreError: dumping log to /tmp/lustre-log.1584557988.20883 [ 3415.167060] LNet: Service thread pid 20995 was inactive for 200.31s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 3415.183993] Pid: 20995, comm: mdt00_024 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [ 3415.194168] Call Trace: [ 3415.196636] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [ 3415.203239] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [ 3415.209910] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [ 3415.216576] [] osp_md_object_lock+0x162/0x2d0 [osp] [ 3415.223136] [] lod_object_lock+0xf4/0x780 [lod] [ 3415.229352] [] mdd_object_lock+0x3e/0xe0 [mdd] [ 3415.235495] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [ 3415.242758] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [ 3415.249501] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [ 3415.255726] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [ 3415.262223] [] mdt_reint_rec+0x83/0x210 [mdt] [ 3415.268275] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 3415.274849] [] mdt_reint+0x67/0x140 [mdt] [ 3415.280556] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 3415.287512] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3415.295229] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 3415.301554] [] kthread+0xd1/0xe0 [ 3415.306470] [] ret_from_fork_nospec_begin+0xe/0x21 [ 3415.312935] [] 0xffffffffffffffff [ 3415.317975] Pid: 20879, comm: mdt03_010 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [ 3415.328147] Call Trace: [ 3415.330603] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [ 3415.337194] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [ 3415.343892] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [ 3415.350565] [] osp_md_object_lock+0x162/0x2d0 [osp] [ 3415.357144] [] lod_object_lock+0xf4/0x780 [lod] [ 3415.363360] [] mdd_object_lock+0x3e/0xe0 [mdd] [ 3415.369501] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [ 3415.376766] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [ 3415.383526] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [ 3415.389755] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [ 3415.396249] [] mdt_reint_rec+0x83/0x210 [mdt] [ 3415.402290] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 3415.408866] [] mdt_reint+0x67/0x140 [mdt] [ 3415.414562] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 3415.421521] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3415.429238] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 3415.435563] [] kthread+0xd1/0xe0 [ 3415.440470] [] ret_from_fork_nospec_begin+0xe/0x21 [ 3415.446934] [] 0xffffffffffffffff [ 3418.751126] LNet: Service thread pid 20959 was inactive for 200.53s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 3418.768065] LNet: Skipped 1 previous similar message [ 3418.773040] Pid: 20959, comm: mdt03_018 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [ 3418.783229] Call Trace: [ 3418.785689] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [ 3418.792283] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [ 3418.798973] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [ 3418.805639] [] osp_md_object_lock+0x162/0x2d0 [osp] [ 3418.812200] [] lod_object_lock+0xf4/0x780 [lod] [ 3418.818414] [] mdd_object_lock+0x3e/0xe0 [mdd] [ 3418.824558] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [ 3418.831823] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [ 3418.838565] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [ 3418.844788] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [ 3418.851271] [] mdt_reint_rec+0x83/0x210 [mdt] [ 3418.857320] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 3418.863896] [] mdt_reint+0x67/0x140 [mdt] [ 3418.869592] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 3418.876550] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3418.884268] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 3418.890606] [] kthread+0xd1/0xe0 [ 3418.895524] [] ret_from_fork_nospec_begin+0xe/0x21 [ 3418.901997] [] 0xffffffffffffffff [ 3418.907036] LustreError: dumping log to /tmp/lustre-log.1584557992.20959 [ 3419.775155] Pid: 20990, comm: mdt00_021 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [ 3419.785330] Call Trace: [ 3419.787795] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [ 3419.794396] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [ 3419.801087] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [ 3419.807751] [] osp_md_object_lock+0x162/0x2d0 [osp] [ 3419.814344] [] lod_object_lock+0xf4/0x780 [lod] [ 3419.820561] [] mdd_object_lock+0x3e/0xe0 [mdd] [ 3419.826705] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [ 3419.833969] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [ 3419.840727] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [ 3419.846953] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [ 3419.853434] [] mdt_reint_rec+0x83/0x210 [mdt] [ 3419.859485] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 3419.866060] [] mdt_reint+0x67/0x140 [mdt] [ 3419.871765] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 3419.878737] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3419.886455] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 3419.892797] [] kthread+0xd1/0xe0 [ 3419.897713] [] ret_from_fork_nospec_begin+0xe/0x21 [ 3419.904204] [] 0xffffffffffffffff [ 3419.909226] LustreError: dumping log to /tmp/lustre-log.1584557993.20990 [ 3419.916421] LNet: Service thread pid 21599 was inactive for 200.71s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 3421.823202] LNet: Service thread pid 20977 was inactive for 200.55s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 3421.836067] LNet: Skipped 5 previous similar messages [ 3421.841128] LustreError: dumping log to /tmp/lustre-log.1584557995.20977 [ 3435.135522] LNet: Service thread pid 21007 was inactive for 200.41s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 3435.148383] LNet: Skipped 9 previous similar messages [ 3435.153444] LustreError: dumping log to /tmp/lustre-log.1584558008.21007 [ 3436.262379] Lustre: fir-MDT0002: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [ 3437.184570] LNet: Service thread pid 20999 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 3437.197434] LustreError: dumping log to /tmp/lustre-log.1584558010.20999 [ 3438.207598] LustreError: dumping log to /tmp/lustre-log.1584558011.20855 [ 3441.791683] LNet: Service thread pid 20867 was inactive for 200.38s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 3441.804550] LNet: Skipped 1 previous similar message [ 3441.809523] LustreError: dumping log to /tmp/lustre-log.1584558015.20867 [ 3446.528881] LustreError: 167-0: fir-MDT0000-lwp-MDT0002: This client was evicted by fir-MDT0000; in progress operations using this service will fail. [ 3448.447846] LustreError: dumping log to /tmp/lustre-log.1584558022.21585 [ 3449.178136] LustreError: 11-0: fir-MDT0000-lwp-MDT0002: operation quota_acquire to node 10.0.10.52@o2ib7 failed: rc = -11 [ 3449.189095] LustreError: Skipped 21 previous similar messages [ 3456.640048] LNet: Service thread pid 21032 was inactive for 200.45s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 3456.652912] LNet: Skipped 1 previous similar message [ 3456.657886] LustreError: dumping log to /tmp/lustre-log.1584558030.21032 [ 3472.000430] LustreError: dumping log to /tmp/lustre-log.1584558045.20993 [ 3477.657560] LNetError: 20288:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [ 3477.667738] LNetError: 20288:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (34): c: 7, oc: 0, rc: 8 [ 3477.679931] LNetError: 20297:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 [ 3478.221751] LNetError: 22194:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3479.221719] LNetError: 22194:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3479.233667] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584558045/real 1584558052] req@ffff8b8b49667080 x1661526579956992/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1584558052 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [ 3479.262067] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 3480.222870] LNetError: 22194:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3482.160448] Lustre: fir-MDT0000-osp-MDT0002: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [ 3482.170106] Lustre: Skipped 1 previous similar message [ 3482.176216] LNet: Service thread pid 21579 completed after 225.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [ 3482.192391] LNet: Skipped 21 previous similar messages [ 3529.346223] LNetError: 22194:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3529.358137] LNetError: 22194:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message [ 3535.230118] LNetError: 22194:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3535.242031] LNetError: 22194:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [ 3579.523302] LNetError: 22194:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3636.661487] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [ 3636.671590] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3636.683506] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [ 3641.661605] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [ 3642.662635] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 3648.661786] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 3685.662668] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 3685.672793] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3685.684728] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [ 3698.662989] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [ 3698.673075] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 2 previous similar messages [ 3740.664046] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 3761.664543] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3761.676449] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 4 previous similar messages [ 3773.664846] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 3773.674926] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 2 previous similar messages [ 3840.666488] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 3840.676568] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 2 previous similar messages [ 3891.667711] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 3891.679624] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 7 previous similar messages [ 4090.672562] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 4090.682645] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 7 previous similar messages [ 4161.675278] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 4161.687196] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 14 previous similar messages [ 4362.679236] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 4362.689322] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 11 previous similar messages [ 4676.686892] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 4676.698808] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 24 previous similar messages [ 5179.235419] LustreError: 11-0: fir-MDT0000-osp-MDT0002: operation ldlm_enqueue to node 10.0.10.52@o2ib7 failed: rc = -107 [ 5179.246374] LustreError: Skipped 42 previous similar messages [ 5179.252132] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 5185.194691] Lustre: fir-MDT0000-lwp-MDT0002: Connection to fir-MDT0000 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 5224.985221] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.2.66@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.002507] LustreError: Skipped 1006 previous similar messages [ 5240.273464] Lustre: fir-MDT0002: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 5240.989434] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.7.8@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.006648] LustreError: Skipped 840 previous similar messages [ 5310.638468] LustreError: 167-0: fir-MDT0000-lwp-MDT0002: This client was evicted by fir-MDT0000; in progress operations using this service will fail. [ 5310.660997] Lustre: fir-MDT0000-lwp-MDT0002: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 5316.261924] LustreError: 11-0: fir-MDT0000-lwp-MDT0002: operation quota_acquire to node 10.0.10.51@o2ib7 failed: rc = -11 [ 5335.082885] Lustre: fir-MDT0000-osp-MDT0002: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 5360.814730] Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x65ffc3416bb7af1a to 0x22d279060839d0ec [ 6383.751883] LustreError: 11-0: fir-OST0000-osc-MDT0002: operation ost_statfs to node 10.0.10.101@o2ib7 failed: rc = -107 [ 6383.762758] Lustre: fir-OST0000-osc-MDT0002: Connection to fir-OST0000 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6384.551929] LustreError: 11-0: fir-OST0004-osc-MDT0002: operation ost_statfs to node 10.0.10.101@o2ib7 failed: rc = -107 [ 6384.562800] LustreError: Skipped 4 previous similar messages [ 6384.568475] Lustre: fir-OST0004-osc-MDT0002: Connection to fir-OST0004 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6384.584566] Lustre: Skipped 4 previous similar messages [ 6408.824471] LustreError: 11-0: fir-OST0018-osc-MDT0002: operation ost_statfs to node 10.0.10.105@o2ib7 failed: rc = -107 [ 6408.835346] Lustre: fir-OST0018-osc-MDT0002: Connection to fir-OST0018 (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6439.625344] Lustre: fir-OST0032-osc-MDT0002: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6439.641413] Lustre: Skipped 5 previous similar messages [ 6439.730128] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.101@o2ib7: 4 seconds [ 6439.740297] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 12 previous similar messages [ 6439.749699] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 6443.849390] LustreError: 11-0: fir-OST0034-osc-MDT0002: operation ost_statfs to node 10.0.10.109@o2ib7 failed: rc = -107 [ 6443.860262] LustreError: Skipped 5 previous similar messages [ 6443.865935] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6450.625410] Lustre: 20310:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561017/real 1584561017] req@ffff8b7b7792a400 x1661526663045376/t0(0) o13->fir-OST003a-osc-MDT0002@10.0.10.109@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584561024 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 6451.193419] Lustre: 20318:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561017/real 1584561017] req@ffff8b7b582d9f80 x1661526663044672/t0(0) o13->fir-OST0038-osc-MDT0002@10.0.10.109@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584561024 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 6451.221544] Lustre: 20318:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6452.065439] Lustre: fir-OST0030-osc-MDT0002: Connection to fir-OST0030 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6452.081508] Lustre: Skipped 3 previous similar messages [ 6463.302200] Lustre: fir-MDT0002: Client 51620f49-4677-4 (at 10.50.15.2@o2ib2) reconnecting [ 6463.310488] Lustre: fir-MDT0002: Connection restored to 51620f49-4677-4 (at 10.50.15.2@o2ib2) [ 6463.319013] Lustre: Skipped 1 previous similar message [ 6466.776858] Lustre: fir-MDT0002: Client 63f6d78f-f55a-4 (at 10.50.6.21@o2ib2) reconnecting [ 6469.497623] Lustre: fir-MDT0002: Client ea174402-2c74-4 (at 10.50.2.16@o2ib2) reconnecting [ 6469.505898] Lustre: Skipped 1 previous similar message [ 6469.511066] Lustre: fir-MDT0002: Connection restored to ea174402-2c74-4 (at 10.50.2.16@o2ib2) [ 6469.519592] Lustre: Skipped 2 previous similar messages [ 6470.718892] Lustre: 20877:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561037/real 1584561037] req@ffff8babaafd8d80 x1661526663067008/t0(0) o104->fir-MDT0002@10.50.8.67@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584561044 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 6470.746139] Lustre: 20877:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6472.290799] Lustre: fir-MDT0002: Client 3e1a7dd1-f48f-4 (at 10.50.2.17@o2ib2) reconnecting [ 6472.299067] Lustre: Skipped 5 previous similar messages [ 6478.607137] Lustre: fir-MDT0002: Client 7cd4fcba-858e-4 (at 10.50.10.54@o2ib2) reconnecting [ 6478.615490] Lustre: Skipped 2 previous similar messages [ 6478.620746] Lustre: fir-MDT0002: Connection restored to 7cd4fcba-858e-4 (at 10.50.10.54@o2ib2) [ 6478.629358] Lustre: Skipped 8 previous similar messages [ 6484.602324] LustreError: 11-0: fir-OST004e-osc-MDT0002: operation ost_statfs to node 10.0.10.113@o2ib7 failed: rc = -107 [ 6484.613200] Lustre: fir-OST004e-osc-MDT0002: Connection to fir-OST004e (at 10.0.10.113@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6486.660641] Lustre: fir-MDT0002: Client e09578c7-8ddc-4 (at 10.50.7.63@o2ib2) reconnecting [ 6486.668911] Lustre: Skipped 5 previous similar messages [ 6493.733465] Lustre: 21578:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561060/real 1584561060] req@ffff8b8b3d303f00 x1661526663094976/t0(0) o104->fir-MDT0002@10.50.16.17@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584561067 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 6493.760830] Lustre: 21578:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6494.699419] Lustre: fir-MDT0002: Connection restored to 95e66b92-7a58-4 (at 10.50.5.22@o2ib2) [ 6494.707945] Lustre: Skipped 17 previous similar messages [ 6499.680790] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.9.50@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.698067] LustreError: Skipped 537 previous similar messages [ 6500.770628] Lustre: 21578:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561067/real 1584561067] req@ffff8b8b3d303f00 x1661526663094976/t0(0) o104->fir-MDT0002@10.50.16.17@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584561074 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [ 6502.851691] Lustre: fir-MDT0002: Client ce7c8b0a-ace3-4 (at 10.50.7.39@o2ib2) reconnecting [ 6502.859957] Lustre: Skipped 65 previous similar messages [ 6503.731717] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.101@o2ib7: 119 seconds [ 6503.742062] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 40 previous similar messages [ 6507.267716] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.49.21.35@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6507.285083] LustreError: Skipped 1 previous similar message [ 6509.094834] Lustre: 20970:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561075/real 1584561075] req@ffff8b7b35f89200 x1661526663105088/t0(0) o104->fir-MDT0002@10.50.2.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584561082 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [ 6509.122089] Lustre: 20970:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6516.943197] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.30.1@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.960480] LustreError: Skipped 3 previous similar messages [ 6527.146853] Lustre: fir-MDT0002: Connection restored to 237d2964-9692-4 (at 10.50.10.65@o2ib2) [ 6527.155475] Lustre: Skipped 134 previous similar messages [ 6528.798317] Lustre: 21578:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561095/real 1584561095] req@ffff8b8b3d303f00 x1661526663094976/t0(0) o104->fir-MDT0002@10.50.16.17@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584561102 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [ 6528.825657] Lustre: 21578:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6533.068770] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.9.19@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.086051] LustreError: Skipped 11 previous similar messages [ 6534.677480] LNetError: 20305:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.224@o2ib7 added to recovery queue. Health = 900 [ 6534.677488] LustreError: 21015:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4096) req@ffff8b7abef74380 x1659201048478656/t0(0) o37->1f9d385a-f33f-4@10.50.6.65@o2ib2:357/0 lens 448/440 e 3 to 0 dl 1584561137 ref 1 fl Interpret:/0/0 rc 0/0 [ 6535.037674] Lustre: fir-MDT0002: Client 249b852d-19fa-4 (at 10.49.25.32@o2ib1) reconnecting [ 6535.046031] Lustre: Skipped 96 previous similar messages [ 6539.732600] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 6539.744517] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 17 previous similar messages [ 6563.836178] Lustre: 21578:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561130/real 1584561130] req@ffff8b8b3d303f00 x1661526663094976/t0(0) o104->fir-MDT0002@10.50.16.17@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584561137 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [ 6563.863520] Lustre: 21578:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 6565.147932] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.10.31@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6565.165308] LustreError: Skipped 56 previous similar messages [ 6591.556320] Lustre: fir-MDT0002: Connection restored to 21098921-f7b5-4 (at 10.50.4.6@o2ib2) [ 6591.564771] Lustre: Skipped 302 previous similar messages [ 6599.593302] Lustre: fir-MDT0002: Client 96da097e-1470-4 (at 10.50.3.55@o2ib2) reconnecting [ 6599.601570] Lustre: Skipped 365 previous similar messages [ 6629.172936] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.2.46@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.190222] LustreError: Skipped 2114 previous similar messages [ 6631.734855] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.105@o2ib7: 1 seconds [ 6631.745024] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 178 previous similar messages [ 6690.427877] Lustre: fir-MDT0002: haven't heard from client 5b88c4a2-39df-4 (at 10.50.17.45@o2ib2) in 194 seconds. I think it's dead, and I am evicting it. exp ffff8babb1fa8400, cur 1584561264 expire 1584561114 last 1584561070 [ 6695.736435] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 6695.748347] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 12 previous similar messages [ 6709.716794] LNetError: 20305:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.225@o2ib7 added to recovery queue. Health = 900 [ 6709.716803] LustreError: 21939:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4096) req@ffff8b8b192d9b00 x1659192242308096/t0(0) o37->0c3d5f54-b140-4@10.50.10.39@o2ib2:510/0 lens 448/440 e 3 to 0 dl 1584561290 ref 1 fl Interpret:/0/0 rc 0/0 [ 6723.552345] Lustre: fir-MDT0002: Connection restored to 99ec6091-5082-4 (at 10.50.8.69@o2ib2) [ 6723.560873] Lustre: Skipped 1094 previous similar messages [ 6727.940032] Lustre: fir-MDT0002: Client b80d20c4-a8bc-4 (at 10.49.26.7@o2ib1) reconnecting [ 6727.948299] Lustre: Skipped 996 previous similar messages [ 6764.369203] LustreError: 11-0: fir-OST0016-osc-MDT0002: operation ost_statfs to node 10.0.10.103@o2ib7 failed: rc = -107 [ 6764.380071] LustreError: Skipped 5 previous similar messages [ 6764.385741] Lustre: fir-OST0016-osc-MDT0002: Connection to fir-OST0016 (at 10.0.10.103@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6764.401822] Lustre: Skipped 5 previous similar messages [ 6804.882210] LustreError: 11-0: fir-OST002e-osc-MDT0002: operation ost_statfs to node 10.0.10.107@o2ib7 failed: rc = -107 [ 6804.893081] LustreError: Skipped 5 previous similar messages [ 6834.578974] Lustre: fir-OST0046-osc-MDT0002: Connection to fir-OST0046 (at 10.0.10.111@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 6834.595050] Lustre: Skipped 12 previous similar messages [ 6841.811014] Lustre: 20319:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561408/real 1584561408] req@ffff8b7ab3ebf080 x1661526663546816/t0(0) o13->fir-OST0044-osc-MDT0002@10.0.10.111@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584561415 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 6841.839135] Lustre: 20319:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 6874.611925] LustreError: 11-0: fir-OST0054-osc-MDT0002: operation ost_statfs to node 10.0.10.115@o2ib7 failed: rc = -107 [ 6874.622796] LustreError: Skipped 4 previous similar messages [ 6884.749768] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.9.41@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.767049] LustreError: Skipped 237 previous similar messages [ 6894.741361] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.103@o2ib7: 12 seconds [ 6894.751621] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 157 previous similar messages [ 6948.943677] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 6948.957498] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584561222, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8baaf923c800/0x2f21cf15387c6be0 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060a25f156 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 6948.994835] LustreError: 40445:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab3a711800) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6950.704728] LustreError: 20953:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b8babf99f80 x1659165303496832/t0(0) o37->c404f1c3-1f95-4@10.50.5.17@o2ib2:18/0 lens 448/440 e 1 to 0 dl 1584561553 ref 1 fl Interpret:/0/0 rc 0/0 [ 6970.854230] Lustre: 20974:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561537/real 1584561537] req@ffff8baaf4685100 x1661526663663104/t0(0) o104->fir-MDT0002@10.50.1.72@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584561544 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [ 6970.881474] Lustre: 20974:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 83 previous similar messages [ 6972.452005] Lustre: fir-MDT0002: haven't heard from client 430e4894-d38d-4 (at 10.50.14.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb1712000, cur 1584561546 expire 1584561396 last 1584561319 [ 6972.471982] Lustre: Skipped 5 previous similar messages [ 6979.839496] Lustre: fir-MDT0002: Connection restored to eea958b7-0ad4-4 (at 10.49.17.24@o2ib1) [ 6979.848110] Lustre: Skipped 243 previous similar messages [ 6984.839674] Lustre: fir-MDT0002: Client a060d9ec-f76c-4 (at 10.49.26.24@o2ib1) reconnecting [ 6984.848025] Lustre: Skipped 212 previous similar messages [ 7000.743988] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [ 7000.755894] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 8 previous similar messages [ 7017.587386] LustreError: 21115:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b98ebadd100 x1659103582483776/t0(0) o37->e1c903ac-38f7-4@10.49.8.20@o2ib1:61/0 lens 448/440 e 0 to 0 dl 1584561596 ref 1 fl Interpret:/0/0 rc 0/0 [ 7047.761133] LustreError: 20500:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.50.5.13@o2ib2 ns: mdt-fir-MDT0002_UUID lock: ffff8b99953f1b00/0x2f21cf153343f111 lrc: 3/0,0 mode: PR/PR res: [0x2c0037822:0x12032:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.50.5.13@o2ib2 remote: 0x23fe8e8b97ea68b4 expref: 2732 pid: 20872 timeout: 7047 lvb_type: 0 [ 7056.448077] Lustre: fir-MDT0002: haven't heard from client 0b19cba1-6454-4 (at 10.50.1.6@o2ib2) in 201 seconds. I think it's dead, and I am evicting it. exp ffff8babaaa84c00, cur 1584561630 expire 1584561480 last 1584561429 [ 7056.467877] Lustre: Skipped 4 previous similar messages [ 7132.422270] Lustre: fir-MDT0002: haven't heard from client e47fbe1a-9f59-4 (at 10.50.4.17@o2ib2) in 184 seconds. I think it's dead, and I am evicting it. exp ffff8babad2db800, cur 1584561706 expire 1584561556 last 1584561522 [ 7132.442152] Lustre: Skipped 20 previous similar messages [ 7208.420657] Lustre: fir-MDT0002: haven't heard from client a52f1005-c8d6-4 (at 10.50.1.15@o2ib2) in 193 seconds. I think it's dead, and I am evicting it. exp ffff8babb6992c00, cur 1584561782 expire 1584561632 last 1584561589 [ 7256.003295] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 7256.017120] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584561529, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b9924be1200/0x2f21cf153885996a lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060a4a53d2 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 7256.054461] LustreError: 40745:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b9a51650840) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7564.382923] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 7564.396758] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584561837, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab21cb6300/0x2f21cf15388be9f3 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060b891406 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 7564.434097] LustreError: 40849:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab1607a180) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7564.453637] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 7564.462939] Lustre: Skipped 1259 previous similar messages [ 7870.322562] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 7870.336394] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584562143, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab31b4a400/0x2f21cf1538928124 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060bb22ca4 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 7870.373755] LustreError: 40951:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8babaeda4900) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8179.922296] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 8179.936124] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584562453, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab032d8b40/0x2f21cf1538951394 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060bd74d2a expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 8179.973498] LustreError: 41050:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab4b9f6f00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8179.993061] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 8180.002372] Lustre: Skipped 1 previous similar message [ 8485.172926] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 8485.186755] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584562758, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab318f9440/0x2f21cf1538976647 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060bfc9063 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 8485.224092] LustreError: 41155:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8baba4f07d40) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8521.827943] INFO: task mdt00_000:20504 blocked for more than 120 seconds. [ 8521.834739] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8521.842577] mdt00_000 D ffff8b9ba0680000 0 20504 2 0x00000080 [ 8521.849679] Call Trace: [ 8521.852150] [] ? lquota_disk_read+0xf2/0x390 [lquota] [ 8521.858875] [] schedule+0x29/0x70 [ 8521.863843] [] rwsem_down_write_failed+0x225/0x3a0 [ 8521.870308] [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] [ 8521.876862] [] call_rwsem_down_write_failed+0x17/0x30 [ 8521.883571] [] down_write+0x2d/0x3d [ 8521.888746] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [ 8521.895459] [] lod_qos_prep_create+0x16a/0x1890 [lod] [ 8521.902177] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [ 8521.908658] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [ 8521.915801] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [ 8521.923471] [] lod_prepare_create+0x215/0x2e0 [lod] [ 8521.930028] [] lod_declare_striped_create+0x1ee/0x980 [lod] [ 8521.937260] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [ 8521.944233] [] lod_declare_create+0x204/0x590 [lod] [ 8521.950791] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [ 8521.958646] [] mdd_declare_create+0x4c/0xdf0 [mdd] [ 8521.965096] [] mdd_create+0x867/0x14a0 [mdd] [ 8521.971051] [] mdt_reint_open+0x224f/0x3240 [mdt] [ 8521.977437] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [ 8521.984932] [] mdt_reint_rec+0x83/0x210 [mdt] [ 8521.990968] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 8521.997503] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [ 8522.004568] [] mdt_intent_open+0x82/0x3a0 [mdt] [ 8522.010789] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 8522.017932] [] mdt_intent_policy+0x435/0xd80 [mdt] [ 8522.024388] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [ 8522.031580] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [ 8522.038288] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [ 8522.045427] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [ 8522.051832] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [ 8522.058915] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [ 8522.066451] [] tgt_enqueue+0x62/0x210 [ptlrpc] [ 8522.072587] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 8522.079517] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 8522.087091] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [ 8522.094171] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8522.101875] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [ 8522.108670] [] ? __wake_up+0x44/0x50 [ 8522.113931] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 8522.120248] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 8522.127647] [] kthread+0xd1/0xe0 [ 8522.132535] [] ? insert_kthread_work+0x40/0x40 [ 8522.138653] [] ret_from_fork_nospec_begin+0xe/0x21 [ 8522.145100] [] ? insert_kthread_work+0x40/0x40 [ 8522.151230] INFO: task mdt01_004:20841 blocked for more than 120 seconds. [ 8522.158054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8522.165885] mdt01_004 D ffff8b8bb4e96180 0 20841 2 0x00000080 [ 8522.173005] Call Trace: [ 8522.175466] [] ? lquota_disk_read+0xf2/0x390 [lquota] [ 8522.182171] [] schedule+0x29/0x70 [ 8522.187160] [] rwsem_down_write_failed+0x225/0x3a0 [ 8522.193607] [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] [ 8522.200137] [] call_rwsem_down_write_failed+0x17/0x30 [ 8522.206860] [] down_write+0x2d/0x3d [ 8522.212010] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [ 8522.218727] [] lod_qos_prep_create+0x16a/0x1890 [lod] [ 8522.225454] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [ 8522.231928] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [ 8522.239079] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [ 8522.246761] [] lod_prepare_create+0x215/0x2e0 [lod] [ 8522.253296] [] lod_declare_striped_create+0x1ee/0x980 [lod] [ 8522.260527] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [ 8522.267519] [] lod_declare_create+0x204/0x590 [lod] [ 8522.274058] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [ 8522.281898] [] mdd_declare_create+0x4c/0xdf0 [mdd] [ 8522.288375] [] mdd_create+0x867/0x14a0 [mdd] [ 8522.294316] [] mdt_reint_open+0x224f/0x3240 [mdt] [ 8522.300693] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [ 8522.308204] [] mdt_reint_rec+0x83/0x210 [mdt] [ 8522.314227] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 8522.320773] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [ 8522.327852] [] mdt_intent_open+0x82/0x3a0 [mdt] [ 8522.334059] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 8522.341211] [] mdt_intent_policy+0x435/0xd80 [mdt] [ 8522.347681] [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] [ 8522.354846] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [ 8522.362018] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [ 8522.368746] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [ 8522.375890] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [ 8522.382288] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [ 8522.389366] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [ 8522.396886] [] tgt_enqueue+0x62/0x210 [ptlrpc] [ 8522.403034] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 8522.409944] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 8522.417539] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [ 8522.424636] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8522.432328] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [ 8522.439134] [] ? __wake_up+0x44/0x50 [ 8522.444390] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 8522.450692] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 8522.458107] [] kthread+0xd1/0xe0 [ 8522.462996] [] ? insert_kthread_work+0x40/0x40 [ 8522.469099] [] ret_from_fork_nospec_begin+0xe/0x21 [ 8522.475560] [] ? insert_kthread_work+0x40/0x40 [ 8522.481658] INFO: task mdt02_009:20872 blocked for more than 120 seconds. [ 8522.488450] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8522.496303] mdt02_009 D ffff8b8bb695c100 0 20872 2 0x00000080 [ 8522.503404] Call Trace: [ 8522.505866] [] ? lquota_disk_read+0xf2/0x390 [lquota] [ 8522.512570] [] schedule+0x29/0x70 [ 8522.517565] [] rwsem_down_write_failed+0x225/0x3a0 [ 8522.524017] [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] [ 8522.530547] [] call_rwsem_down_write_failed+0x17/0x30 [ 8522.537270] [] down_write+0x2d/0x3d [ 8522.542419] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [ 8522.549136] [] lod_qos_prep_create+0x16a/0x1890 [lod] [ 8522.555867] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [ 8522.562317] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [ 8522.569470] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [ 8522.577154] [] lod_prepare_create+0x215/0x2e0 [lod] [ 8522.583689] [] lod_declare_striped_create+0x1ee/0x980 [lod] [ 8522.590927] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [ 8522.597916] [] lod_declare_create+0x204/0x590 [lod] [ 8522.604455] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [ 8522.612297] [] mdd_declare_create+0x4c/0xdf0 [mdd] [ 8522.618773] [] mdd_create+0x867/0x14a0 [mdd] [ 8522.624706] [] mdt_reint_open+0x224f/0x3240 [mdt] [ 8522.631082] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [ 8522.638595] [] mdt_reint_rec+0x83/0x210 [mdt] [ 8522.644609] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 8522.651155] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [ 8522.658232] [] mdt_intent_open+0x82/0x3a0 [mdt] [ 8522.664430] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 8522.671583] [] mdt_intent_policy+0x435/0xd80 [mdt] [ 8522.678053] [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] [ 8522.685212] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [ 8522.692376] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [ 8522.699102] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [ 8522.706248] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [ 8522.712640] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [ 8522.719736] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [ 8522.727257] [] tgt_enqueue+0x62/0x210 [ptlrpc] [ 8522.733394] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 8522.740323] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 8522.747898] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [ 8522.754979] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8522.762682] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [ 8522.769476] [] ? __wake_up+0x44/0x50 [ 8522.774738] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 8522.781056] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 8522.788455] [] kthread+0xd1/0xe0 [ 8522.793345] [] ? insert_kthread_work+0x40/0x40 [ 8522.799460] [] ret_from_fork_nospec_begin+0xe/0x21 [ 8522.805901] [] ? insert_kthread_work+0x40/0x40 [ 8522.812006] INFO: task mdt00_008:20883 blocked for more than 120 seconds. [ 8522.818828] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8522.826659] mdt00_008 D ffff8b8bb0a0c100 0 20883 2 0x00000080 [ 8522.833754] Call Trace: [ 8522.836230] [] ? lquota_disk_read+0xf2/0x390 [lquota] [ 8522.842937] [] schedule+0x29/0x70 [ 8522.847925] [] rwsem_down_write_failed+0x225/0x3a0 [ 8522.854373] [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] [ 8522.860904] [] call_rwsem_down_write_failed+0x17/0x30 [ 8522.867627] [] down_write+0x2d/0x3d [ 8522.872774] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [ 8522.879492] [] lod_qos_prep_create+0x16a/0x1890 [lod] [ 8522.886222] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [ 8522.892674] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [ 8522.899824] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [ 8522.907513] [] lod_prepare_create+0x215/0x2e0 [lod] [ 8522.914053] [] lod_declare_striped_create+0x1ee/0x980 [lod] [ 8522.921284] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [ 8522.928275] [] lod_declare_create+0x204/0x590 [lod] [ 8522.934811] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [ 8522.942645] [] mdd_declare_create+0x4c/0xdf0 [mdd] [ 8522.949121] [] mdd_create+0x867/0x14a0 [mdd] [ 8522.955053] [] mdt_reint_open+0x224f/0x3240 [mdt] [ 8522.961430] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [ 8522.968942] [] mdt_reint_rec+0x83/0x210 [mdt] [ 8522.974958] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 8522.981503] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [ 8522.988580] [] mdt_intent_open+0x82/0x3a0 [mdt] [ 8522.994779] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 8523.001931] [] mdt_intent_policy+0x435/0xd80 [mdt] [ 8523.008402] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [ 8523.015577] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [ 8523.022287] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [ 8523.029450] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [ 8523.035836] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [ 8523.042920] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [ 8523.050452] [] tgt_enqueue+0x62/0x210 [ptlrpc] [ 8523.056582] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 8523.063489] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 8523.071087] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [ 8523.078167] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8523.085858] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [ 8523.092668] [] ? __wake_up+0x44/0x50 [ 8523.097926] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 8523.104229] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 8523.111643] [] kthread+0xd1/0xe0 [ 8523.116525] [] ? insert_kthread_work+0x40/0x40 [ 8523.122629] [] ret_from_fork_nospec_begin+0xe/0x21 [ 8523.129087] [] ? insert_kthread_work+0x40/0x40 [ 8523.135200] INFO: task mdt00_028:21004 blocked for more than 120 seconds. [ 8523.141996] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8523.149864] mdt00_028 D ffff8b7b9d050000 0 21004 2 0x00000080 [ 8523.156961] Call Trace: [ 8523.159421] [] ? lquota_disk_read+0xf2/0x390 [lquota] [ 8523.166125] [] schedule+0x29/0x70 [ 8523.171116] [] rwsem_down_write_failed+0x225/0x3a0 [ 8523.177562] [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] [ 8523.184092] [] call_rwsem_down_write_failed+0x17/0x30 [ 8523.190815] [] down_write+0x2d/0x3d [ 8523.195961] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [ 8523.202672] [] lod_qos_prep_create+0x16a/0x1890 [lod] [ 8523.209415] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [ 8523.215876] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [ 8523.223022] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [ 8523.230706] [] lod_prepare_create+0x215/0x2e0 [lod] [ 8523.237243] [] lod_declare_striped_create+0x1ee/0x980 [lod] [ 8523.244473] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [ 8523.251463] [] lod_declare_create+0x204/0x590 [lod] [ 8523.258000] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [ 8523.265834] [] mdd_declare_create+0x4c/0xdf0 [mdd] [ 8523.272297] [] mdd_create+0x867/0x14a0 [mdd] [ 8523.278249] [] mdt_reint_open+0x224f/0x3240 [mdt] [ 8523.284630] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [ 8523.292140] [] mdt_reint_rec+0x83/0x210 [mdt] [ 8523.298156] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 8523.304702] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [ 8523.311780] [] mdt_intent_open+0x82/0x3a0 [mdt] [ 8523.317978] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 8523.325128] [] mdt_intent_policy+0x435/0xd80 [mdt] [ 8523.331597] [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] [ 8523.338742] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [ 8523.345911] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [ 8523.352639] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [ 8523.359783] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [ 8523.366168] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [ 8523.373267] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [ 8523.380785] [] tgt_enqueue+0x62/0x210 [ptlrpc] [ 8523.386922] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 8523.393852] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 8523.401427] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [ 8523.408507] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8523.416224] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [ 8523.423011] [] ? __wake_up+0x44/0x50 [ 8523.428276] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 8523.434593] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 8523.441993] [] kthread+0xd1/0xe0 [ 8523.446883] [] ? finish_task_switch+0x54/0x1c0 [ 8523.452997] [] ? insert_kthread_work+0x40/0x40 [ 8523.459094] [] ret_from_fork_nospec_begin+0xe/0x21 [ 8523.465542] [] ? insert_kthread_work+0x40/0x40 [ 8523.471670] INFO: task mdt02_040:21579 blocked for more than 120 seconds. [ 8523.478476] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8523.486308] mdt02_040 D ffff8b7b566c2080 0 21579 2 0x00000080 [ 8523.493427] Call Trace: [ 8523.495887] [] ? lquota_disk_read+0xf2/0x390 [lquota] [ 8523.502594] [] schedule+0x29/0x70 [ 8523.507582] [] rwsem_down_write_failed+0x225/0x3a0 [ 8523.514031] [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] [ 8523.520569] [] call_rwsem_down_write_failed+0x17/0x30 [ 8523.527292] [] down_write+0x2d/0x3d [ 8523.532447] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [ 8523.539164] [] lod_qos_prep_create+0x16a/0x1890 [lod] [ 8523.545894] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [ 8523.552347] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [ 8523.559499] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [ 8523.567183] [] lod_prepare_create+0x215/0x2e0 [lod] [ 8523.573718] [] lod_declare_striped_create+0x1ee/0x980 [lod] [ 8523.580949] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [ 8523.587939] [] lod_declare_create+0x204/0x590 [lod] [ 8523.594475] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [ 8523.602311] [] mdd_declare_create+0x4c/0xdf0 [mdd] [ 8523.608780] [] mdd_create+0x867/0x14a0 [mdd] [ 8523.614718] [] mdt_reint_open+0x224f/0x3240 [mdt] [ 8523.621096] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [ 8523.628609] [] mdt_reint_rec+0x83/0x210 [mdt] [ 8523.634632] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 8523.641177] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [ 8523.648256] [] mdt_intent_open+0x82/0x3a0 [mdt] [ 8523.654462] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 8523.661612] [] mdt_intent_policy+0x435/0xd80 [mdt] [ 8523.668085] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [ 8523.675258] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [ 8523.681968] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [ 8523.689132] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [ 8523.695518] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [ 8523.702601] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [ 8523.710134] [] tgt_enqueue+0x62/0x210 [ptlrpc] [ 8523.716263] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [ 8523.723170] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 8523.730760] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [ 8523.737841] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8523.745530] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [ 8523.752335] [] ? __wake_up+0x44/0x50 [ 8523.757592] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [ 8523.763897] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 8523.771314] [] kthread+0xd1/0xe0 [ 8523.776200] [] ? insert_kthread_work+0x40/0x40 [ 8523.782299] [] ret_from_fork_nospec_begin+0xe/0x21 [ 8523.788763] [] ? insert_kthread_work+0x40/0x40 [ 8795.183636] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 8795.197463] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584563068, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab20ba0b40/0x2f21cf15389914ff lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060c247c0f expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 8795.234876] LustreError: 41364:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8baba4f06240) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8795.254416] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 8795.263718] Lustre: Skipped 6 previous similar messages [ 9102.684273] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 9102.698099] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584563376, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8baba8365a00/0x2f21cf1538a66709 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060d4e7ca1 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 9102.735507] LustreError: 41619:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab43788fc0) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9412.875830] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 9412.889655] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584563686, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b7b8dd5cec0/0x2f21cf1539134416 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060e79253a expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 9412.927040] LustreError: 41711:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab375740c0) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9412.946582] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 9412.955897] Lustre: Skipped 3 previous similar messages [ 9721.867408] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 9721.881237] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584563995, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab20371440/0x2f21cf15398d871c lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060eeeb02d expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [ 9721.918635] LustreError: 41825:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab3da9e300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9739.232837] LustreError: 11-0: fir-OST0020-osc-MDT0002: operation ost_destroy to node 10.0.10.106@o2ib7 failed: rc = -19 [ 9739.232839] LustreError: 11-0: fir-OST0020-osc-MDT0002: operation ost_destroy to node 10.0.10.106@o2ib7 failed: rc = -19 [ 9739.232842] LustreError: Skipped 5 previous similar messages [ 9739.232847] Lustre: fir-OST0020-osc-MDT0002: Connection to fir-OST0020 (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 9739.232848] Lustre: Skipped 10 previous similar messages [ 9739.281651] LustreError: Skipped 3 previous similar messages [ 9754.699298] LustreError: 11-0: fir-OST0046-osc-MDT0002: operation ost_statfs to node 10.0.10.112@o2ib7 failed: rc = -107 [ 9754.710171] LustreError: Skipped 32 previous similar messages [ 9756.075340] Lustre: fir-OST0008-osc-MDT0002: Connection to fir-OST0008 (at 10.0.10.102@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 9756.091408] Lustre: Skipped 27 previous similar messages [ 9941.395282] LustreError: 11-0: fir-OST001b-osc-MDT0002: operation ost_create to node 10.0.10.106@o2ib7 failed: rc = -107 [ 9941.406155] LustreError: Skipped 23 previous similar messages [ 9941.411932] Lustre: fir-OST001b-osc-MDT0002: Connection to fir-OST001b (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 9941.428019] Lustre: Skipped 19 previous similar messages [ 9941.433360] LustreError: 20651:0:(osp_precreate.c:686:osp_precreate_send()) fir-OST001b-osc-MDT0002: can't precreate: rc = -107 [ 9953.584108] Lustre: 20315:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584564520/real 1584564520] req@ffff8b7b1de86780 x1661526685416192/t0(0) o13->fir-OST000f-osc-MDT0002@10.0.10.104@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584564527 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 9953.612226] Lustre: 20315:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 9970.572093] Lustre: fir-MDT0002: Client 7b634cda-eb45-4 (at 10.50.6.32@o2ib2) reconnecting [ 9970.580361] Lustre: Skipped 1186 previous similar messages [ 9994.241105] Lustre: 20343:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584564523/real 1584564523] req@ffff8b98ccf58480 x1661526685420544/t0(0) o6->fir-OST002b-osc-MDT0002@10.0.10.107@o2ib7:28/4 lens 544/432 e 0 to 1 dl 1584564567 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 9994.269228] Lustre: 20343:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [10008.818495] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.102@o2ib7: 4 seconds [10008.828663] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 212 previous similar messages [10008.838158] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [10008.850101] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [10015.631743] Lustre: fir-MDT0002: Connection restored to 4171bc70-1760-4 (at 10.50.6.27@o2ib2) [10015.640278] Lustre: Skipped 153 previous similar messages [10017.915516] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.7.14@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [10017.932804] LustreError: Skipped 2689 previous similar messages [10028.418945] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [10028.432770] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584564301, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8baaf9ad1d40/0x2f21cf153a3b22f5 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060f589d46 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [10028.470138] LustreError: 41942:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab265d46c0) refcount nonzero (1) after lock cleanup; forcing cleanup. [10035.919937] Lustre: fir-MDT0002: Client 6eb34c30-3804-4 (at 10.49.19.6@o2ib1) reconnecting [10035.928208] Lustre: Skipped 9 previous similar messages [10050.633301] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.14.15@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [10050.650684] LustreError: Skipped 28 previous similar messages [10114.647327] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.29.4@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.664609] LustreError: Skipped 159 previous similar messages [10128.634408] LustreError: 21015:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b7b90e3c050 x1659238479833600/t0(0) o37->50a6df69-dfc3-4@10.50.12.6@o2ib2:161/0 lens 448/440 e 1 to 0 dl 1584564716 ref 1 fl Interpret:/2/0 rc 0/0 [10163.934523] Lustre: fir-MDT0002: Client f3d4dc47-07cf-4 (at 10.50.10.42@o2ib2) reconnecting [10163.942883] Lustre: Skipped 960 previous similar messages [10165.563588] Lustre: fir-MDT0002: haven't heard from client fb2c1382-8f5a-4 (at 10.50.15.10@o2ib2) in 226 seconds. I think it's dead, and I am evicting it. exp ffff8babb5368000, cur 1584564739 expire 1584564589 last 1584564513 [10643.334043] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [10643.347859] LustreError: Skipped 1 previous similar message [10643.353454] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584564916, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b7b083998c0/0x2f21cf153aa72c88 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906100c2b44 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [10643.390580] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [10643.401558] LustreError: 42163:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b8b14442480) refcount nonzero (1) after lock cleanup; forcing cleanup. [10643.421095] LustreError: 42163:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [10643.431537] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [10643.440858] Lustre: Skipped 1756 previous similar messages [11260.287203] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [11260.301014] LustreError: Skipped 1 previous similar message [11260.306607] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584565533, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab264d1440/0x2f21cf153bffce4a lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790610e5ae7d expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [11260.343775] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [11260.354766] LustreError: 42381:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b7ad8107740) refcount nonzero (1) after lock cleanup; forcing cleanup. [11260.374303] LustreError: 42381:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [11260.384743] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [11260.394054] Lustre: Skipped 1 previous similar message [11877.392323] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [11877.406134] LustreError: Skipped 1 previous similar message [11877.411733] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584566150, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab1fed4c80/0x2f21cf153da9e0e4 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790611bbdef0 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [11877.448882] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [11877.459905] LustreError: 42583:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b987475cc00) refcount nonzero (1) after lock cleanup; forcing cleanup. [11877.479441] LustreError: 42583:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [11877.489892] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [11877.499202] Lustre: Skipped 1 previous similar message [12146.444025] LustreError: 11-0: fir-OST004b-osc-MDT0002: operation ost_destroy to node 10.0.10.113@o2ib7 failed: rc = -107 [12146.444035] Lustre: fir-OST004b-osc-MDT0002: Connection to fir-OST004b (at 10.0.10.113@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [12146.444038] Lustre: Skipped 47 previous similar messages [12146.476382] LustreError: Skipped 69 previous similar messages [12150.790132] LustreError: 11-0: fir-OST0009-osc-MDT0002: operation ost_statfs to node 10.0.10.101@o2ib7 failed: rc = -107 [12150.801006] LustreError: Skipped 31 previous similar messages [12488.376274] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [12488.390088] LustreError: Skipped 1 previous similar message [12488.395684] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584566761, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b7b98e49d40/0x2f21cf153f1cdca0 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906127f74d1 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [12488.432817] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [12488.443828] LustreError: 42999:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b7b37ab0b40) refcount nonzero (1) after lock cleanup; forcing cleanup. [12488.463381] LustreError: 42999:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [12488.473826] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [12488.483133] Lustre: Skipped 97 previous similar messages [13102.402020] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [13102.415840] LustreError: Skipped 1 previous similar message [13102.421433] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584567375, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b98cc251680/0x2f21cf153fc66df3 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790613007c49 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [13102.458560] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [13102.469556] LustreError: 43202:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab4daad8c0) refcount nonzero (1) after lock cleanup; forcing cleanup. [13102.489092] LustreError: 43202:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [13102.499531] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [13102.508832] Lustre: Skipped 1 previous similar message [13719.328011] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [13719.341827] LustreError: Skipped 1 previous similar message [13719.347419] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584567992, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab3963af40/0x2f21cf1540917000 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790613bd35ee expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [13719.384600] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [13719.395603] LustreError: 43415:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8baad4a2b200) refcount nonzero (1) after lock cleanup; forcing cleanup. [13719.415136] LustreError: 43415:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [13719.425577] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [13719.434896] Lustre: Skipped 1 previous similar message [14333.796111] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [14333.809931] LustreError: Skipped 1 previous similar message [14333.815523] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584568607, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b7aafeff740/0x2f21cf154179aeaa lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906147870c3 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [14333.852670] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [14333.863673] LustreError: 43630:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab02bc8cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. [14333.883230] LustreError: 43630:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [14333.893670] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [14333.902977] Lustre: Skipped 1 previous similar message [14947.604199] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [14947.618020] LustreError: Skipped 1 previous similar message [14947.623613] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584569220, 301s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8baacc3ca400/0x2f21cf1542902de9 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906153c2e9e expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [14947.660739] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [14947.671776] LustreError: 43834:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b7b356a2600) refcount nonzero (1) after lock cleanup; forcing cleanup. [14947.691333] LustreError: 43834:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [14947.701812] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [14947.711122] Lustre: Skipped 1 previous similar message [15563.134171] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [15563.147987] LustreError: Skipped 1 previous similar message [15563.153583] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584569836, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab1fa5e300/0x2f21cf1543b679fb lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906160a2fb7 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [15563.190752] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [15563.201761] LustreError: 44039:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b7ac28e9440) refcount nonzero (1) after lock cleanup; forcing cleanup. [15563.221305] LustreError: 44039:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [15563.231745] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [15563.241059] Lustre: Skipped 1 previous similar message [16178.534831] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [16178.548647] LustreError: Skipped 1 previous similar message [16178.554241] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584570451, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab0d708fc0/0x2f21cf15454b632f lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790616e7abb7 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [16178.591385] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [16178.602397] LustreError: 44236:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab1801bc80) refcount nonzero (1) after lock cleanup; forcing cleanup. [16178.621935] LustreError: 44236:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [16178.632377] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [16178.641687] Lustre: Skipped 1 previous similar message [16797.498632] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [16797.512455] LustreError: Skipped 1 previous similar message [16797.518044] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584571070, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b7a4fa26c00/0x2f21cf15466c6291 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906179b47fb expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [16797.555208] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [16797.566216] LustreError: 44438:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b7add3a6600) refcount nonzero (1) after lock cleanup; forcing cleanup. [16797.585777] LustreError: 44438:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [16797.596234] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [16797.605548] Lustre: Skipped 1 previous similar message [16916.655015] Lustre: fir-MDT0002: haven't heard from client 3c0f2777-beb3-4 (at 10.50.1.60@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba4c4fbd400, cur 1584571490 expire 1584571340 last 1584571263 [16916.674901] Lustre: Skipped 59 previous similar messages [17410.851266] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [17410.865079] LustreError: Skipped 1 previous similar message [17410.870674] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584571684, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b7a4fa08000/0x2f21cf1547eb0477 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061855a22f expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [17410.907831] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [17410.918843] LustreError: 44643:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8baab06c3200) refcount nonzero (1) after lock cleanup; forcing cleanup. [17410.938399] LustreError: 44643:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [17410.948857] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [17410.958167] Lustre: Skipped 1 previous similar message [18028.355987] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [18028.369797] LustreError: Skipped 1 previous similar message [18028.375392] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584572301, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8baa9abf86c0/0x2f21cf1549089743 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790618eeabae expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [18028.412537] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [18028.423555] LustreError: 44858:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8bab2bb22000) refcount nonzero (1) after lock cleanup; forcing cleanup. [18028.443098] LustreError: 44858:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [18028.453538] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [18028.462849] Lustre: Skipped 1 previous similar message [18118.015138] LNetError: 20288:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [18118.025311] LNetError: 20288:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (105): c: 7, oc: 0, rc: 8 [18238.687785] Lustre: fir-MDT0002: haven't heard from client f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb6b17000, cur 1584572812 expire 1584572662 last 1584572585 [18645.940821] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [18645.954637] LustreError: Skipped 1 previous similar message [18645.960234] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584572919, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8bab423b33c0/0x2f21cf154a3c038d lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906198b1678 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [18645.997392] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [18646.008412] LustreError: 45062:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8baabbf90540) refcount nonzero (1) after lock cleanup; forcing cleanup. [18646.027963] LustreError: 45062:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [18646.038420] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [18646.047738] Lustre: Skipped 2 previous similar messages [19259.924469] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [19259.938283] LustreError: Skipped 1 previous similar message [19259.943876] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584573533, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b7a5381b840/0x2f21cf154b4343dc lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061a3aac57 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [19259.981023] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [19259.992061] LustreError: 45255:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b7b1b32ab40) refcount nonzero (1) after lock cleanup; forcing cleanup. [19260.011600] LustreError: 45255:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [19260.022037] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [19260.031339] Lustre: Skipped 1 previous similar message [19874.900194] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [19874.914011] LustreError: Skipped 1 previous similar message [19874.919601] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574148, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8baa7aefec00/0x2f21cf154c0849c9 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061afab2f6 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [19874.956762] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [19874.967779] LustreError: 45470:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8baaa2ec2d80) refcount nonzero (1) after lock cleanup; forcing cleanup. [19874.987317] LustreError: 45470:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [19874.997768] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [19875.007077] Lustre: Skipped 1 previous similar message [20488.306893] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [20488.320713] LustreError: Skipped 1 previous similar message [20488.326306] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574761, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8baaeaa1ec00/0x2f21cf154cdb7866 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061bce0b2a expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [20488.363459] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [20488.374508] LustreError: 45667:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b98cd787440) refcount nonzero (1) after lock cleanup; forcing cleanup. [20488.394065] LustreError: 45667:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [20488.404527] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [20488.413839] Lustre: Skipped 1 previous similar message [20657.743249] Lustre: fir-MDT0002: haven't heard from client 15c6dd40-b461-4 (at 10.50.1.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb0e09000, cur 1584575231 expire 1584575081 last 1584575004 [20977.011613] Lustre: 20355:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584575543/real 1584575543] req@ffff8bab1c234800 x1661526879971584/t0(0) o400->fir-MDT0000-lwp-MDT0002@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1584575550 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [20977.013613] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [20977.013614] Lustre: Skipped 47 previous similar messages [20977.061299] Lustre: 20355:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [20981.083708] LNetError: 20288:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [20981.093709] LNetError: 20288:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (6): c: 0, oc: 0, rc: 8 [20981.105873] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [20982.083743] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [20982.093826] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 143 previous similar messages [20994.084019] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [20994.094103] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 6 previous similar messages [20994.103418] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [20994.115338] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 4 previous similar messages [21027.222565] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.0.61@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [21027.239849] LustreError: Skipped 3573 previous similar messages [21033.084962] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [21033.095070] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [21043.422002] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.19.5@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [21043.439279] LustreError: Skipped 155 previous similar messages [21077.443602] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.27@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [21077.460889] LustreError: Skipped 142 previous similar messages [21079.086074] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [21079.096162] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 2 previous similar messages [21079.105483] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [21079.117393] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [21099.895571] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584575373, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8baa6e229200/0x2f21cf154e20676c lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061c8066a9 expref: -99 pid: 20501 timeout: 0 lvb_type: 0 [21099.932700] LustreError: 20501:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [21141.553202] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.8.29@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [21141.570480] LustreError: Skipped 500 previous similar messages [21146.087683] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [21146.097766] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 10 previous similar messages [21171.247273] LNet: Service thread pid 41527 was inactive for 200.61s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [21171.264232] LNet: Skipped 1 previous similar message [21171.269217] Pid: 41527, comm: mdt03_069 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [21171.279428] Call Trace: [21171.281899] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [21171.288517] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [21171.295224] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [21171.301898] [] osp_md_object_lock+0x162/0x2d0 [osp] [21171.308469] [] lod_object_lock+0xf4/0x780 [lod] [21171.314700] [] mdd_object_lock+0x3e/0xe0 [mdd] [21171.320845] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [21171.328133] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [21171.334901] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [21171.341151] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [21171.347666] [] mdt_reint_rec+0x83/0x210 [mdt] [21171.353734] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [21171.360307] [] mdt_reint+0x67/0x140 [mdt] [21171.366033] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [21171.372995] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [21171.380759] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [21171.387104] [] kthread+0xd1/0xe0 [21171.392025] [] ret_from_fork_nospec_begin+0xe/0x21 [21171.398544] [] 0xffffffffffffffff [21171.403589] LustreError: dumping log to /tmp/lustre-log.1584575744.41527 [21171.432136] Pid: 41484, comm: mdt00_056 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [21171.442317] Call Trace: [21171.444789] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [21171.451421] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [21171.458137] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [21171.464828] [] osp_md_object_lock+0x162/0x2d0 [osp] [21171.471399] [] lod_object_lock+0xf4/0x780 [lod] [21171.477630] [] mdd_object_lock+0x3e/0xe0 [mdd] [21171.483776] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [21171.491060] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [21171.497807] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [21171.504061] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [21171.510548] [] mdt_reint_rec+0x83/0x210 [mdt] [21171.516618] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [21171.523192] [] mdt_reint+0x67/0x140 [mdt] [21171.528917] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [21171.535879] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [21171.543626] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [21171.549965] [] kthread+0xd1/0xe0 [21171.554907] [] ret_from_fork_nospec_begin+0xe/0x21 [21171.561413] [] 0xffffffffffffffff [21172.783304] LNet: Service thread pid 41513 was inactive for 200.51s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [21172.800239] LNet: Skipped 1 previous similar message [21172.805212] Pid: 41513, comm: mdt01_074 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [21172.815400] Call Trace: [21172.817862] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [21172.824473] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [21172.831148] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [21172.837820] [] osp_md_object_lock+0x162/0x2d0 [osp] [21172.844405] [] lod_object_lock+0xf4/0x780 [lod] [21172.850631] [] mdd_object_lock+0x3e/0xe0 [mdd] [21172.856766] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [21172.864029] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [21172.870772] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [21172.876996] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [21172.883493] [] mdt_reint_rec+0x83/0x210 [mdt] [21172.889544] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [21172.896114] [] mdt_reint+0x67/0x140 [mdt] [21172.901817] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [21172.908768] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [21172.916483] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [21172.922825] [] kthread+0xd1/0xe0 [21172.927741] [] ret_from_fork_nospec_begin+0xe/0x21 [21172.934212] [] 0xffffffffffffffff [21172.939234] LustreError: dumping log to /tmp/lustre-log.1584575746.41513 [21174.319341] LNet: Service thread pid 41472 was inactive for 200.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [21174.336274] Pid: 41472, comm: mdt00_052 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [21174.346447] Call Trace: [21174.348936] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [21174.355535] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [21174.362223] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [21174.368908] [] osp_md_object_lock+0x162/0x2d0 [osp] [21174.375469] [] lod_object_lock+0xf4/0x780 [lod] [21174.381725] [] mdd_object_lock+0x3e/0xe0 [mdd] [21174.387853] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [21174.395116] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [21174.401859] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [21174.408096] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [21174.414584] [] mdt_reint_rec+0x83/0x210 [mdt] [21174.420652] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [21174.427218] [] mdt_reint+0x67/0x140 [mdt] [21174.432922] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [21174.439888] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [21174.447629] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [21174.453957] [] kthread+0xd1/0xe0 [21174.458887] [] ret_from_fork_nospec_begin+0xe/0x21 [21174.465376] [] 0xffffffffffffffff [21174.470415] LustreError: dumping log to /tmp/lustre-log.1584575747.41472 [21174.477681] Pid: 20871, comm: mdt01_009 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [21174.487876] Call Trace: [21174.490341] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [21174.496927] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [21174.503632] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [21174.510300] [] osp_md_object_lock+0x162/0x2d0 [osp] [21174.516877] [] lod_object_lock+0xf4/0x780 [lod] [21174.523092] [] mdd_object_lock+0x3e/0xe0 [mdd] [21174.529253] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [21174.536543] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [21174.543302] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [21174.549559] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [21174.556058] [] mdt_reint_rec+0x83/0x210 [mdt] [21174.562111] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [21174.568685] [] mdt_reint+0x67/0x140 [mdt] [21174.574384] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [21174.581349] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [21174.589065] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [21174.595414] [] kthread+0xd1/0xe0 [21174.600325] [] ret_from_fork_nospec_begin+0xe/0x21 [21174.606813] [] 0xffffffffffffffff [21176.367392] LNet: Service thread pid 20509 was inactive for 200.37s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [21176.380251] LNet: Skipped 18 previous similar messages [21176.385400] LustreError: dumping log to /tmp/lustre-log.1584575749.20509 [21182.088543] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [21182.100449] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 8 previous similar messages [21191.727760] LNet: Service thread pid 21586 was inactive for 200.54s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [21191.740624] LustreError: dumping log to /tmp/lustre-log.1584575764.21586 [21196.847881] LNet: Service thread pid 41478 was inactive for 200.03s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [21196.860759] LustreError: dumping log to /tmp/lustre-log.1584575770.41478 [21218.352394] LNet: Service thread pid 20988 was inactive for 200.68s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [21218.365259] LNet: Skipped 7 previous similar messages [21218.370319] LustreError: dumping log to /tmp/lustre-log.1584575791.20988 [21220.912460] LustreError: dumping log to /tmp/lustre-log.1584575794.21577 [21226.544589] LustreError: dumping log to /tmp/lustre-log.1584575799.21011 [21227.568614] LustreError: dumping log to /tmp/lustre-log.1584575800.41349 [21229.616664] LustreError: dumping log to /tmp/lustre-log.1584575802.21597 [21233.712762] LustreError: dumping log to /tmp/lustre-log.1584575806.41334 [21236.784836] LNet: Service thread pid 41320 was inactive for 200.29s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [21236.797702] LNet: Skipped 13 previous similar messages [21236.802852] LustreError: dumping log to /tmp/lustre-log.1584575810.41320 [21237.808862] LustreError: dumping log to /tmp/lustre-log.1584575811.20959 [21269.655705] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.52@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [21269.672987] LustreError: Skipped 1436 previous similar messages [21343.368065] Lustre: fir-MDT0002: Received LWP connection from 10.0.10.52@o2ib7, removing former export from 10.0.10.51@o2ib7 [21343.379375] Lustre: fir-MDT0002: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [21343.388011] Lustre: Skipped 2 previous similar messages [21380.148251] LNet: Service thread pid 41514 was inactive for 286.59s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [21380.161112] LNet: Skipped 1 previous similar message [21380.166104] LustreError: dumping log to /tmp/lustre-log.1584575953.41514 [21526.763984] Lustre: fir-MDT0002: haven't heard from client 48a86d34-282f-4 (at 10.50.5.38@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb7b5f800, cur 1584576100 expire 1584575950 last 1584575873 [21530.679847] LNet: Service thread pid 41493 was inactive for 387.20s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [21530.696778] LNet: Skipped 1 previous similar message [21530.701756] Pid: 41493, comm: mdt01_066 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [21530.711943] Call Trace: [21530.714406] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [21530.721018] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [21530.727705] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [21530.734381] [] osp_md_object_lock+0x162/0x2d0 [osp] [21530.740943] [] lod_object_lock+0xf4/0x780 [lod] [21530.747182] [] mdd_object_lock+0x3e/0xe0 [mdd] [21530.753333] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [21530.760600] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [21530.767356] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [21530.773585] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [21530.780084] [] mdt_reint_rec+0x83/0x210 [mdt] [21530.786131] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [21530.792719] [] mdt_reint+0x67/0x140 [mdt] [21530.798420] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [21530.805388] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [21530.813113] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [21530.819454] [] kthread+0xd1/0xe0 [21530.824385] [] ret_from_fork_nospec_begin+0xe/0x21 [21530.830878] [] 0xffffffffffffffff [21530.835908] LustreError: dumping log to /tmp/lustre-log.1584576104.41493 [21561.400576] LNet: Service thread pid 20985 was inactive for 412.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [21561.417513] Pid: 20985, comm: mdt03_025 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [21561.427699] Call Trace: [21561.430161] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [21561.436762] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [21561.443436] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [21561.450125] [] osp_md_object_lock+0x162/0x2d0 [osp] [21561.456689] [] lod_object_lock+0xf4/0x780 [lod] [21561.462902] [] mdd_object_lock+0x3e/0xe0 [mdd] [21561.469039] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [21561.476316] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [21561.483062] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [21561.489299] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [21561.495785] [] mdt_reint_rec+0x83/0x210 [mdt] [21561.501834] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [21561.508407] [] mdt_reint+0x67/0x140 [mdt] [21561.514131] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [21561.521076] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [21561.528789] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [21561.535117] [] kthread+0xd1/0xe0 [21561.540044] [] ret_from_fork_nospec_begin+0xe/0x21 [21561.546538] [] 0xffffffffffffffff [21561.551583] LustreError: dumping log to /tmp/lustre-log.1584576134.20985 [21564.984671] Lustre: 41495:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b7ba35cda00 x1659584382216640/t0(0) o36->20f8c0d3-7223-4@10.50.5.51@o2ib2:263/0 lens 576/2888 e 24 to 0 dl 1584576143 ref 2 fl Interpret:/0/0 rc 0/0 [21566.570712] Lustre: 41340:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b7830d3c800 x1659661233033216/t0(0) o36->a5f3f76c-fda8-4@10.50.8.3@o2ib2:264/0 lens 568/2888 e 24 to 0 dl 1584576144 ref 2 fl Interpret:/0/0 rc 0/0 [21568.577757] Lustre: 41490:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b782ca9de80 x1659159732799616/t0(0) o36->5fa9fd8f-5c3b-4@10.50.6.25@o2ib2:266/0 lens 568/2888 e 23 to 0 dl 1584576146 ref 2 fl Interpret:/0/0 rc 0/0 [21568.604769] Lustre: 41490:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [21569.592774] LNet: Service thread pid 20920 was inactive for 400.20s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [21569.609714] Pid: 20920, comm: mdt02_013 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [21569.619887] Call Trace: [21569.622352] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [21569.628954] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [21569.635628] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [21569.642318] [] osp_md_object_lock+0x162/0x2d0 [osp] [21569.648879] [] lod_object_lock+0xf4/0x780 [lod] [21569.655134] [] mdd_object_lock+0x3e/0xe0 [mdd] [21569.661272] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [21569.668550] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [21569.675312] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [21569.681551] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [21569.688035] [] mdt_reint_rec+0x83/0x210 [mdt] [21569.694101] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [21569.700672] [] mdt_reint+0x67/0x140 [mdt] [21569.706390] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [21569.713344] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [21569.721067] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [21569.727392] [] kthread+0xd1/0xe0 [21569.732322] [] ret_from_fork_nospec_begin+0xe/0x21 [21569.738799] [] 0xffffffffffffffff [21569.743832] LustreError: dumping log to /tmp/lustre-log.1584576142.20920 [21570.860815] Lustre: 20989:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b8a3befde80 x1659149503188736/t0(0) o36->16466280-91fc-4@10.50.6.13@o2ib2:269/0 lens 568/2888 e 23 to 0 dl 1584576149 ref 2 fl Interpret:/0/0 rc 0/0 [21570.887806] Lustre: 20989:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [21571.693954] Lustre: fir-MDT0002: Client 20f8c0d3-7223-4 (at 10.50.5.51@o2ib2) reconnecting [21571.702231] Lustre: Skipped 684 previous similar messages [21592.057322] Lustre: 41540:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8baaeee9c380 x1659546475930176/t0(0) o36->9fd8baa5-4600-4@10.50.6.15@o2ib2:290/0 lens 552/2888 e 12 to 0 dl 1584576170 ref 2 fl Interpret:/0/0 rc 0/0 [21604.282797] LustreError: 167-0: fir-MDT0000-lwp-MDT0002: This client was evicted by fir-MDT0000; in progress operations using this service will fail. [21612.089795] Lustre: 41540:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8baa713cda00 x1659299694649344/t0(0) o36->4f300b23-846a-4@10.50.1.1@o2ib2:310/0 lens 568/2888 e 10 to 0 dl 1584576190 ref 2 fl Interpret:/0/0 rc 0/0 [21612.116697] Lustre: 41540:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages [21618.698576] Lustre: fir-MDT0002: Client 4f300b23-846a-4 (at 10.50.1.1@o2ib2) reconnecting [21618.706755] Lustre: Skipped 13 previous similar messages [21628.698197] Lustre: 40617:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b7ab0e22880 x1659551949685376/t0(0) o36->48495bd3-7f9d-4@10.50.17.48@o2ib2:326/0 lens 552/2888 e 7 to 0 dl 1584576206 ref 2 fl Interpret:/0/0 rc 0/0 [21628.725187] Lustre: 40617:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages [21629.370495] Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x22d279060839d0ec to 0xae6d2c631bd4526c [21629.382525] LustreError: 46105:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8baa6c703080) refcount nonzero (1) after lock cleanup; forcing cleanup. [21629.402100] LustreError: 46105:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [21688.731638] Lustre: 41490:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b7af83c0000 x1659403951630976/t0(0) o36->5b88c4a2-39df-4@10.50.17.45@o2ib2:386/0 lens 552/2888 e 4 to 0 dl 1584576266 ref 2 fl Interpret:/0/0 rc 0/0 [21688.758632] Lustre: 41490:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [21694.622992] Lustre: fir-MDT0002: Client 5b88c4a2-39df-4 (at 10.50.17.45@o2ib2) reconnecting [21694.631346] Lustre: Skipped 13 previous similar messages [21893.696578] Lustre: 41326:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b8a34605a00 x1660724499595584/t0(0) o36->8c27500b-ffc7-4@10.50.0.64@o2ib2:591/0 lens 536/2888 e 1 to 0 dl 1584576471 ref 2 fl Interpret:/0/0 rc 0/0 [21893.723486] Lustre: 41326:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [21899.189078] LNet: Service thread pid 21586 completed after 907.98s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [21899.197717] Lustre: 41477:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8b8a34605a00 x1660724499595584/t493955241463(0) o36->8c27500b-ffc7-4@10.50.0.64@o2ib2:591/0 lens 536/424 e 1 to 0 dl 1584576471 ref 1 fl Complete:/0/0 rc 0/0 [21899.232710] LNet: Skipped 35 previous similar messages [24089.539574] perf: interrupt took too long (2507 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [30631.119022] perf: interrupt took too long (3185 > 3133), lowering kernel.perf_event_max_sample_rate to 62000 [40493.835359] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584595059/real 1584595059] req@ffff8b7badab1b00 x1661526968442240/t0(0) o104->fir-MDT0002@10.50.9.23@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584595066 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [40493.862613] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [40500.872526] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584595066/real 1584595066] req@ffff8b7badab1b00 x1661526968442240/t0(0) o104->fir-MDT0002@10.50.9.23@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584595073 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [40507.900702] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584595073/real 1584595073] req@ffff8b7badab1b00 x1661526968442240/t0(0) o104->fir-MDT0002@10.50.9.23@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584595080 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [40514.927880] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584595080/real 1584595080] req@ffff8b7badab1b00 x1661526968442240/t0(0) o104->fir-MDT0002@10.50.9.23@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584595087 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [40528.957229] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584595094/real 1584595094] req@ffff8b7badab1b00 x1661526968442240/t0(0) o104->fir-MDT0002@10.50.9.23@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584595101 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [40528.984503] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [40549.993758] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584595115/real 1584595115] req@ffff8b7badab1b00 x1661526968442240/t0(0) o104->fir-MDT0002@10.50.9.23@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584595122 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [40550.021005] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [40567.232680] Lustre: fir-MDT0002: haven't heard from client ca4d9d7f-c632-4 (at 10.50.9.23@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bababbb8c00, cur 1584595140 expire 1584594990 last 1584594913 [40567.252591] LustreError: 20999:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.9.23@o2ib2) failed to reply to blocking AST (req@ffff8b7badab1b00 x1661526968442240 status 0 rc -5), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8bab15e7d580/0x2f21cf16310278c3 lrc: 4/0,0 mode: PR/PR res: [0x2c0037b19:0xe92:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.50.9.23@o2ib2 remote: 0x18c44ad6235ba1c2 expref: 53994 pid: 20919 timeout: 40712 lvb_type: 0 [40567.295184] LustreError: 138-a: fir-MDT0002: A client on nid 10.50.9.23@o2ib2 was evicted due to a lock blocking callback time out: rc -5 [40616.059480] Lustre: fir-MDT0002: Connection restored to ca4d9d7f-c632-4 (at 10.50.9.23@o2ib2) [40616.068012] Lustre: Skipped 34 previous similar messages [41340.249414] Lustre: fir-MDT0002: haven't heard from client f4363950-d6c3-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaaa86800, cur 1584595913 expire 1584595763 last 1584595686 [41396.928997] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [47599.409688] Lustre: fir-MDT0002: haven't heard from client eceee209-ec05-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb6992000, cur 1584602172 expire 1584602022 last 1584601945 [47675.313966] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [65789.877655] Lustre: fir-MDT0002: haven't heard from client 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba8e49800, cur 1584620362 expire 1584620212 last 1584620135 [65844.706960] Lustre: fir-MDT0002: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [84361.324672] Lustre: fir-MDT0002: haven't heard from client a2a51302-74aa-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab51264c00, cur 1584638933 expire 1584638783 last 1584638706 [84415.953034] Lustre: fir-MDT0002: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [89083.434051] Lustre: fir-MDT0002: haven't heard from client e584cba3-332a-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8730fb5000, cur 1584643655 expire 1584643505 last 1584643428 [89134.288648] Lustre: fir-MDT0002: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [113112.631058] Lustre: fir-MDT0002: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [113112.641150] Lustre: Skipped 1 previous similar message [113112.646410] Lustre: fir-MDT0002: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [128245.145975] Lustre: fir-MDT0002: Connection restored to 4f32e72b-02b6-4 (at 10.50.9.6@o2ib2) [128245.404654] Lustre: fir-MDT0002: haven't heard from client 4f32e72b-02b6-4 (at 10.50.9.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babac1c1800, cur 1584682816 expire 1584682666 last 1584682589 [130735.468163] Lustre: fir-MDT0002: haven't heard from client 417f1855-bc48-4 (at 10.50.9.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babbf19f800, cur 1584685306 expire 1584685156 last 1584685079 [137409.125563] Lustre: fir-MDT0002: Client 20f8c0d3-7223-4 (at 10.50.5.51@o2ib2) reconnecting [137409.133942] Lustre: fir-MDT0002: Connection restored to 20f8c0d3-7223-4 (at 10.50.5.51@o2ib2) [159261.216269] LustreError: 46738:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.50.9.7@o2ib2 arrived at 1584713831 with bad export cookie 3396223283536289201 [159261.216565] Lustre: fir-MDT0002: Connection restored to 417f1855-bc48-4 (at 10.50.9.7@o2ib2) [159261.240256] LustreError: 46738:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 19 previous similar messages [159488.204459] Lustre: fir-MDT0002: haven't heard from client 417f1855-bc48-4 (at 10.50.9.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7d732c5000, cur 1584714058 expire 1584713908 last 1584713831 [172439.314408] Lustre: fir-MDT0002: Client 42993fde-8354-4 (at 10.50.7.65@o2ib2) reconnecting [172439.322793] Lustre: fir-MDT0002: Connection restored to 42993fde-8354-4 (at 10.50.7.65@o2ib2) [181504.899130] Lustre: fir-MDT0002: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [181504.907508] Lustre: fir-MDT0002: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [181529.801542] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [181529.801543] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [181529.801546] LustreError: Skipped 1374 previous similar messages [181554.890214] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [181655.244729] Lustre: fir-MDT0002: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [181655.253100] Lustre: fir-MDT0002: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [184302.807198] Lustre: fir-MDT0002: haven't heard from client 2bdad291-4dd1-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab58ea9400, cur 1584738872 expire 1584738722 last 1584738645 [184356.255740] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [184640.098399] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [184641.318418] Lustre: fir-MDT0002: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [184641.326796] Lustre: fir-MDT0002: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [184641.579399] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [184722.165342] LustreError: 20500:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.49.0.62@o2ib1 ns: mdt-fir-MDT0002_UUID lock: ffff8b97da768900/0x2f21cf19a75c4f7e lrc: 3/0,0 mode: PR/PR res: [0x2c0039175:0x1fec2:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.49.0.62@o2ib1 remote: 0xea7701772aae1dbd expref: 2542014 pid: 21586 timeout: 184717 lvb_type: 0 [184823.822856] LNet: Service thread pid 20872 was inactive for 200.71s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [184823.839878] Pid: 20872, comm: mdt02_009 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [184823.850141] Call Trace: [184823.852696] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [184823.859730] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [184823.867013] [] mdt_object_local_lock+0x438/0xb20 [mdt] [184823.873937] [] mdt_object_lock_internal+0x70/0x360 [mdt] [184823.881023] [] mdt_object_lock+0x20/0x30 [mdt] [184823.887237] [] mdt_reint_open+0x106a/0x3240 [mdt] [184823.893738] [] mdt_reint_rec+0x83/0x210 [mdt] [184823.899866] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [184823.906514] [] mdt_intent_open+0x82/0x3a0 [mdt] [184823.912813] [] mdt_intent_policy+0x435/0xd80 [mdt] [184823.919382] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [184823.926229] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [184823.933424] [] tgt_enqueue+0x62/0x210 [ptlrpc] [184823.939672] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [184823.946693] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [184823.954496] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [184823.960908] [] kthread+0xd1/0xe0 [184823.965907] [] ret_from_fork_nospec_begin+0xe/0x21 [184823.972470] [] 0xffffffffffffffff [184823.977577] LustreError: dumping log to /tmp/lustre-log.1584739393.20872 [184923.178340] LustreError: 20872:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584739192, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b8fa5a0aac0/0x2f21cf19a9eba8e0 lrc: 3/0,1 mode: --/CW res: [0x2c0039175:0x1fec2:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 20872 timeout: 0 lvb_type: 0 [184923.217960] LustreError: dumping log to /tmp/lustre-log.1584739492.20872 [185134.849718] Lustre: fir-MDT0002: haven't heard from client 22cd96b1-4e55-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9acafa6400, cur 1584739704 expire 1584739554 last 1584739477 [185176.950777] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [185218.768727] Lustre: 41542:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b9ab3abba80 x1660615399559552/t0(0) o101->f7ce444e-0508-4@10.50.12.1@o2ib2:77/0 lens 2872/3288 e 24 to 0 dl 1584739792 ref 2 fl Interpret:/0/0 rc 0/0 [185218.795979] Lustre: 41542:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [185224.185787] Lustre: fir-MDT0002: Client f7ce444e-0508-4 (at 10.50.12.1@o2ib2) reconnecting [185224.194161] Lustre: fir-MDT0002: Connection restored to f7ce444e-0508-4 (at 10.50.12.1@o2ib2) [185382.893274] LNet: Service thread pid 20872 completed after 759.76s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [185382.909527] LNet: Skipped 14 previous similar messages [186507.860841] Lustre: fir-MDT0002: haven't heard from client 4119ac7e-3e26-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b2f762400, cur 1584741077 expire 1584740927 last 1584740850 [186537.725173] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [188269.905283] Lustre: fir-MDT0002: haven't heard from client e1f1914e-b10c-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7f35340400, cur 1584742839 expire 1584742689 last 1584742612 [188321.772806] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [190050.673980] Lustre: fir-MDT0002: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [190279.287759] Lustre: fir-MDT0002: Connection restored to d8ba59b7-d352-4 (at 10.50.10.30@o2ib2) [190286.958143] Lustre: fir-MDT0002: haven't heard from client d8ba59b7-d352-4 (at 10.50.10.30@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb1fad400, cur 1584744856 expire 1584744706 last 1584744629 [191483.985240] Lustre: fir-MDT0002: haven't heard from client 7521def8-3cfd-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b55b6e800, cur 1584746053 expire 1584745903 last 1584745826 [191709.133481] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [197481.133785] Lustre: fir-MDT0002: haven't heard from client 1a29ad5f-9961-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab51262000, cur 1584752050 expire 1584751900 last 1584751823 [197528.902676] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [199010.177900] Lustre: fir-MDT0002: haven't heard from client 34d41e1f-f6fe-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b83252da000, cur 1584753579 expire 1584753429 last 1584753352 [199056.254221] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [200337.203900] Lustre: fir-MDT0002: haven't heard from client 809d0a51-afb6-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b93f4792400, cur 1584754906 expire 1584754756 last 1584754679 [200391.211899] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [205897.024906] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584760458/real 1584760458] req@ffff8ba74caaec00 x1661529191705472/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584760465 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [205897.052250] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [205904.062074] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584760465/real 1584760465] req@ffff8ba74caaec00 x1661529191705472/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584760472 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [205918.089421] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584760479/real 1584760479] req@ffff8ba74caaec00 x1661529191705472/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584760486 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [205918.116763] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [205939.126945] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584760500/real 1584760500] req@ffff8ba74caaec00 x1661529191705472/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584760507 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [205939.154288] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [205974.164818] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584760535/real 1584760535] req@ffff8ba74caaec00 x1661529191705472/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584760542 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [205974.192178] Lustre: 20999:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [205995.202350] LustreError: 20999:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.9.37@o2ib2) failed to reply to blocking AST (req@ffff8ba74caaec00 x1661529191705472 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b7b91399b00/0x2f21cf19cbd9ede6 lrc: 4/0,0 mode: PR/PR res: [0x2c001306c:0x17649:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.50.9.37@o2ib2 remote: 0x880ffd5162a0998d expref: 5113 pid: 41490 timeout: 206082 lvb_type: 0 [205995.245383] LustreError: 138-a: fir-MDT0002: A client on nid 10.50.9.37@o2ib2 was evicted due to a lock blocking callback time out: rc -110 [205995.258009] LustreError: 20500:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.50.9.37@o2ib2 ns: mdt-fir-MDT0002_UUID lock: ffff8b7b91399b00/0x2f21cf19cbd9ede6 lrc: 3/0,0 mode: PR/PR res: [0x2c001306c:0x17649:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.50.9.37@o2ib2 remote: 0x880ffd5162a0998d expref: 5114 pid: 41490 timeout: 0 lvb_type: 0 [206135.956105] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [207968.404434] Lustre: fir-MDT0002: haven't heard from client 83ad7733-be39-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b786fdfb000, cur 1584762537 expire 1584762387 last 1584762310 [208076.710549] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [212755.021893] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.8.58@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [212755.039261] LustreError: Skipped 1 previous similar message [215353.579524] Lustre: fir-MDT0002: haven't heard from client e312085b-9063-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b90967e3000, cur 1584769922 expire 1584769772 last 1584769695 [216514.695614] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [221925.061252] Lustre: fir-MDT0002: Client 203cf276-de8a-4 (at 10.49.7.13@o2ib1) reconnecting [221925.069638] Lustre: fir-MDT0002: Connection restored to 203cf276-de8a-4 (at 10.49.7.13@o2ib1) [222988.769636] Lustre: fir-MDT0002: haven't heard from client 3e58d365-2c0a-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b8ce54400, cur 1584777557 expire 1584777407 last 1584777330 [223027.276121] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [239981.877302] Lustre: fir-MDT0002: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [240041.199112] Lustre: fir-MDT0002: haven't heard from client e71e196d-00dc-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb2de6c00, cur 1584794609 expire 1584794459 last 1584794382 [249895.445818] Lustre: fir-MDT0002: haven't heard from client 55eb8d67-6a00-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b3625e400, cur 1584804463 expire 1584804313 last 1584804236 [249932.447753] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [284752.465670] Lustre: fir-MDT0002: Connection restored to a47ccd42-c337-4 (at 10.50.17.43@o2ib2) [284754.303129] Lustre: fir-MDT0002: haven't heard from client a47ccd42-c337-4 (at 10.50.17.43@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb6b12800, cur 1584839321 expire 1584839171 last 1584839094 [287504.370916] Lustre: fir-MDT0002: haven't heard from client 59748e26-ab50-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb536f000, cur 1584842071 expire 1584841921 last 1584841844 [287537.795798] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [345015.802156] Lustre: fir-MDT0002: haven't heard from client 28cde8ea-b100-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7771557000, cur 1584899581 expire 1584899431 last 1584899354 [345055.374033] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [373244.500038] Lustre: fir-MDT0002: haven't heard from client 1a7a3e3a-e746-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7ba065e000, cur 1584927809 expire 1584927659 last 1584927582 [373283.212786] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [373774.509200] Lustre: fir-MDT0002: haven't heard from client bc12991d-b630-4 (at 10.50.6.22@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babad2dcc00, cur 1584928339 expire 1584928189 last 1584928112 [408158.353466] Lustre: fir-MDT0002: haven't heard from client c6acd997-7594-4 (at 10.50.9.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb9fbfc00, cur 1584962722 expire 1584962572 last 1584962495 [408392.406044] Lustre: fir-MDT0002: Connection restored to c6acd997-7594-4 (at 10.50.9.72@o2ib2) [408619.364602] Lustre: fir-MDT0002: haven't heard from client c6acd997-7594-4 (at 10.50.9.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b52a14400, cur 1584963183 expire 1584963033 last 1584962956 [423350.263683] Lustre: fir-MDT0002: Connection restored to fd4f78ff-b4bc-4 (at 10.50.2.23@o2ib2) [423358.705471] Lustre: fir-MDT0002: Connection restored to 360f9d52-0c81-4 (at 10.50.2.36@o2ib2) [423376.606150] Lustre: fir-MDT0002: Connection restored to 14fe6180-7b80-4 (at 10.49.27.34@o2ib1) [423407.845079] Lustre: fir-MDT0002: Connection restored to 144964dd-491d-4 (at 10.50.4.19@o2ib2) [423414.727275] Lustre: fir-MDT0002: haven't heard from client 144964dd-491d-4 (at 10.50.4.19@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb7b5d800, cur 1584977978 expire 1584977828 last 1584977751 [423416.998091] Lustre: fir-MDT0002: Connection restored to c11f79a1-0fcf-4 (at 10.50.5.1@o2ib2) [423441.390432] Lustre: fir-MDT0002: Connection restored to 05b2fcf3-a8de-4 (at 10.50.8.9@o2ib2) [423441.398967] Lustre: Skipped 3 previous similar messages [423469.970053] Lustre: fir-MDT0002: Connection restored to 857d9564-42d5-4 (at 10.50.9.66@o2ib2) [423469.978668] Lustre: Skipped 1 previous similar message [423513.222404] Lustre: fir-MDT0002: Connection restored to bc12991d-b630-4 (at 10.50.6.22@o2ib2) [423513.231017] Lustre: Skipped 3 previous similar messages [423612.078825] Lustre: fir-MDT0002: Connection restored to 63f2b4c1-0051-4 (at 10.49.8.19@o2ib1) [427341.593729] Lustre: fir-MDT0002: Connection restored to 7afa46a0-f8e2-4 (at 10.50.17.42@o2ib2) [427342.823653] Lustre: fir-MDT0002: haven't heard from client 7afa46a0-f8e2-4 (at 10.50.17.42@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb5087c00, cur 1584981906 expire 1584981756 last 1584981679 [427342.843714] Lustre: Skipped 11 previous similar messages [441196.161997] Lustre: fir-MDT0002: haven't heard from client ce67c0db-bb4c-4 (at 10.50.15.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaaa85400, cur 1584995759 expire 1584995609 last 1584995532 [442557.197936] Lustre: fir-MDT0002: haven't heard from client 2e6641b9-e38a-4 (at 10.50.12.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb536d800, cur 1584997120 expire 1584996970 last 1584996893 [442557.217905] Lustre: Skipped 2 previous similar messages [442633.197490] Lustre: fir-MDT0002: haven't heard from client 6d03d9a7-b1f2-4 (at 10.50.12.9@o2ib2) in 196 seconds. I think it's dead, and I am evicting it. exp ffff8babae6eac00, cur 1584997196 expire 1584997046 last 1584997000 [442643.951884] Lustre: fir-MDT0002: Connection restored to 6d03d9a7-b1f2-4 (at 10.50.12.9@o2ib2) [442970.930990] Lustre: fir-MDT0002: Connection restored to 4304d775-beb4-4 (at 10.50.13.7@o2ib2) [443150.223498] Lustre: fir-MDT0002: haven't heard from client 645377df-0a0a-4 (at 10.50.14.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb298c800, cur 1584997713 expire 1584997563 last 1584997486 [443615.230684] Lustre: fir-MDT0002: Connection restored to f8bb9b7d-b961-4 (at 10.50.14.9@o2ib2) [443701.224475] Lustre: fir-MDT0002: haven't heard from client f8bb9b7d-b961-4 (at 10.50.14.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb5369c00, cur 1584998264 expire 1584998114 last 1584998037 [443811.698917] Lustre: fir-MDT0002: Connection restored to 2e6641b9-e38a-4 (at 10.50.12.8@o2ib2) [444175.202014] Lustre: fir-MDT0002: Connection restored to c05f4f30-e5ee-4 (at 10.50.12.10@o2ib2) [444189.236176] Lustre: fir-MDT0002: haven't heard from client c05f4f30-e5ee-4 (at 10.50.12.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaaa82000, cur 1584998752 expire 1584998602 last 1584998525 [444428.333316] Lustre: fir-MDT0002: Connection restored to 645377df-0a0a-4 (at 10.50.14.10@o2ib2) [446273.758523] Lustre: fir-MDT0002: Connection restored to 50a6df69-dfc3-4 (at 10.50.12.6@o2ib2) [446328.287689] Lustre: fir-MDT0002: haven't heard from client 50a6df69-dfc3-4 (at 10.50.12.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb1711400, cur 1585000891 expire 1585000741 last 1585000664 [446660.685310] Lustre: fir-MDT0002: Connection restored to 2d9a52b2-e42c-4 (at 10.50.12.11@o2ib2) [446670.297409] Lustre: fir-MDT0002: haven't heard from client 2d9a52b2-e42c-4 (at 10.50.12.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb298e000, cur 1585001233 expire 1585001083 last 1585001006 [447463.316835] Lustre: fir-MDT0002: haven't heard from client 2422a71a-3ef8-4 (at 10.50.14.12@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb298d000, cur 1585002026 expire 1585001876 last 1585001799 [447473.911399] Lustre: fir-MDT0002: Connection restored to 8530b1fc-5bf1-4 (at 10.50.14.13@o2ib2) [448104.664671] Lustre: fir-MDT0002: Connection restored to 38135a83-ec5c-4 (at 10.50.12.5@o2ib2) [448180.337464] Lustre: fir-MDT0002: haven't heard from client 38135a83-ec5c-4 (at 10.50.12.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babae6ef000, cur 1585002743 expire 1585002593 last 1585002516 [448180.357441] Lustre: Skipped 2 previous similar messages [448590.684622] Lustre: fir-MDT0002: Connection restored to e63c4438-c6e3-4 (at 10.50.15.8@o2ib2) [448656.706915] Lustre: fir-MDT0002: Connection restored to 72c8f4c4-9808-4 (at 10.50.13.4@o2ib2) [448711.567742] Lustre: fir-MDT0002: Connection restored to 46aae41a-dd38-4 (at 10.50.15.1@o2ib2) [448729.683902] Lustre: fir-MDT0002: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [448746.257600] Lustre: fir-MDT0002: Connection restored to 2d2dde34-cdb2-4 (at 10.50.15.7@o2ib2) [448746.266219] Lustre: Skipped 1 previous similar message [448780.160923] Lustre: fir-MDT0002: Connection restored to d66e7d02-84db-4 (at 10.49.28.11@o2ib1) [448780.169623] Lustre: Skipped 1 previous similar message [448783.350202] Lustre: fir-MDT0002: haven't heard from client fb2c1382-8f5a-4 (at 10.50.15.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ba1f91400, cur 1585003346 expire 1585003196 last 1585003119 [448845.964419] Lustre: fir-MDT0002: Connection restored to ccea6ca8-94f6-4 (at 10.50.15.3@o2ib2) [448845.973032] Lustre: Skipped 2 previous similar messages [449664.615355] Lustre: fir-MDT0002: Connection restored to 0c02c6b3-fbbf-4 (at 10.50.13.14@o2ib2) [449664.624056] Lustre: Skipped 2 previous similar messages [449919.677834] Lustre: fir-MDT0002: Connection restored to fd153b9f-50b1-4 (at 10.50.13.6@o2ib2) [449935.816408] Lustre: fir-MDT0002: Connection restored to 842a3b99-802c-4 (at 10.50.14.7@o2ib2) [449983.699093] Lustre: fir-MDT0002: Connection restored to 7c9c28a0-1550-4 (at 10.50.15.11@o2ib2) [449983.707791] Lustre: Skipped 4 previous similar messages [450718.547588] Lustre: fir-MDT0002: Connection restored to e0b3c403-4bb2-4 (at 10.50.14.6@o2ib2) [450718.556210] Lustre: Skipped 1 previous similar message [451374.753275] Lustre: fir-MDT0002: Connection restored to 0f689ff7-6991-4 (at 10.50.14.4@o2ib2) [451395.337547] Lustre: fir-MDT0002: Connection restored to ef0fa67c-0910-4 (at 10.50.14.2@o2ib2) [452705.579156] Lustre: fir-MDT0002: Connection restored to 482fd27b-a473-4 (at 10.50.13.11@o2ib2) [452734.456420] Lustre: fir-MDT0002: haven't heard from client 482fd27b-a473-4 (at 10.50.13.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba0241400, cur 1585007297 expire 1585007147 last 1585007070 [452734.476472] Lustre: Skipped 11 previous similar messages [453471.469238] Lustre: fir-MDT0002: haven't heard from client e1991612-ee0d-4 (at 10.50.14.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba507297000, cur 1585008034 expire 1585007884 last 1585007807 [453657.469116] Lustre: fir-MDT0002: haven't heard from client fbd4143c-cb04-4 (at 10.50.13.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb922c000, cur 1585008220 expire 1585008070 last 1585007993 [453865.475756] Lustre: fir-MDT0002: haven't heard from client e4a1516e-1b3e-4 (at 10.50.12.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb1fae000, cur 1585008428 expire 1585008278 last 1585008201 [454185.483641] Lustre: fir-MDT0002: haven't heard from client 1ce69533-b12b-4 (at 10.49.29.3@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babae6ea400, cur 1585008748 expire 1585008598 last 1585008521 [454606.493934] Lustre: fir-MDT0002: haven't heard from client 2ac9ee52-f29e-4 (at 10.49.29.1@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb2988800, cur 1585009169 expire 1585009019 last 1585008942 [454681.658213] Lustre: fir-MDT0002: Connection restored to 8530b1fc-5bf1-4 (at 10.50.14.13@o2ib2) [455064.926592] Lustre: fir-MDT0002: Connection restored to e4a1516e-1b3e-4 (at 10.50.12.15@o2ib2) [455065.504874] Lustre: fir-MDT0002: haven't heard from client 4c86e85e-ac4f-4 (at 10.49.28.1@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaa2e7c00, cur 1585009628 expire 1585009478 last 1585009401 [455519.448453] Lustre: fir-MDT0002: Connection restored to 1ce69533-b12b-4 (at 10.49.29.3@o2ib1) [455582.700017] Lustre: fir-MDT0002: Connection restored to fbd4143c-cb04-4 (at 10.50.13.8@o2ib2) [455715.882120] Lustre: fir-MDT0002: Connection restored to 2672fd12-19fb-4 (at 10.50.12.7@o2ib2) [455928.839582] Lustre: fir-MDT0002: Connection restored to 2ac9ee52-f29e-4 (at 10.49.29.1@o2ib1) [456419.805752] Lustre: fir-MDT0002: Connection restored to 4c86e85e-ac4f-4 (at 10.49.28.1@o2ib1) [456857.656075] Lustre: fir-MDT0002: Connection restored to 6627faaf-f7b9-4 (at 10.50.12.17@o2ib2) [458672.928443] Lustre: fir-MDT0002: Connection restored to ab87f0a5-0357-4 (at 10.49.29.5@o2ib1) [459737.366307] Lustre: fir-MDT0002: Connection restored to b8ff2e89-6825-4 (at 10.50.13.2@o2ib2) [459923.626091] Lustre: fir-MDT0002: haven't heard from client 3e79a8aa-e20a-4 (at 10.50.12.16@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb298f800, cur 1585014486 expire 1585014336 last 1585014259 [461113.474306] Lustre: fir-MDT0002: Connection restored to 3e79a8aa-e20a-4 (at 10.50.12.16@o2ib2) [461390.662641] Lustre: fir-MDT0002: haven't heard from client 5d22cf6d-6c39-4 (at 10.49.29.7@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb298c000, cur 1585015953 expire 1585015803 last 1585015726 [462727.380274] Lustre: fir-MDT0002: Connection restored to 5d22cf6d-6c39-4 (at 10.49.29.7@o2ib1) [464376.736407] Lustre: fir-MDT0002: haven't heard from client b4f8cb5a-edfb-4 (at 10.50.13.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb1faa800, cur 1585018939 expire 1585018789 last 1585018712 [465591.937862] Lustre: fir-MDT0002: Connection restored to b4f8cb5a-edfb-4 (at 10.50.13.3@o2ib2) [466190.781719] Lustre: fir-MDT0002: haven't heard from client 35cd2c2f-e02c-4 (at 10.50.12.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaa2e5800, cur 1585020753 expire 1585020603 last 1585020526 [467260.809064] Lustre: fir-MDT0002: haven't heard from client 13489a82-b9be-4 (at 10.50.13.1@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb579cc00, cur 1585021823 expire 1585021673 last 1585021596 [467379.253317] Lustre: fir-MDT0002: Connection restored to 35cd2c2f-e02c-4 (at 10.50.12.13@o2ib2) [468207.830919] Lustre: fir-MDT0002: haven't heard from client 37a9513e-0a78-4 (at 10.50.12.14@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb1fae400, cur 1585022770 expire 1585022620 last 1585022543 [468432.665130] Lustre: fir-MDT0002: Connection restored to 13489a82-b9be-4 (at 10.50.13.1@o2ib2) [468436.837598] Lustre: fir-MDT0002: haven't heard from client 8db8300c-7159-4 (at 10.50.14.14@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb536cc00, cur 1585022999 expire 1585022849 last 1585022772 [468436.857654] Lustre: Skipped 2 previous similar messages [469118.853636] Lustre: fir-MDT0002: haven't heard from client efbc6add-d7ba-4 (at 10.50.12.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb4ccac00, cur 1585023681 expire 1585023531 last 1585023454 [469398.367522] Lustre: fir-MDT0002: Connection restored to 37a9513e-0a78-4 (at 10.50.12.14@o2ib2) [469515.292721] Lustre: fir-MDT0002: Connection restored to 91cab27d-d429-4 (at 10.49.29.2@o2ib1) [469539.059747] Lustre: fir-MDT0002: Connection restored to 2a9a2350-c0db-4 (at 10.49.29.4@o2ib1) [469552.865439] Lustre: fir-MDT0002: haven't heard from client 90f2fa35-5a19-4 (at 10.49.29.8@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaaa86c00, cur 1585024115 expire 1585023965 last 1585023888 [469640.867909] Lustre: fir-MDT0002: Connection restored to 8db8300c-7159-4 (at 10.50.14.14@o2ib2) [471730.090284] Lustre: fir-MDT0002: Connection restored to efbc6add-d7ba-4 (at 10.50.12.4@o2ib2) [473108.954389] Lustre: fir-MDT0002: haven't heard from client 17ee7eca-4216-4 (at 10.50.13.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb536e000, cur 1585027671 expire 1585027521 last 1585027444 [474393.262436] Lustre: fir-MDT0002: Connection restored to 17ee7eca-4216-4 (at 10.50.13.5@o2ib2) [474548.990441] Lustre: fir-MDT0002: haven't heard from client f7fe261e-a413-4 (at 10.49.28.2@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba4c4fbc800, cur 1585029111 expire 1585028961 last 1585028884 [474657.692646] Lustre: fir-MDT0002: Connection restored to d2f2b8c6-0661-4 (at 10.50.16.5@o2ib2) [478496.315189] Lustre: fir-MDT0002: Connection restored to 6c45c03c-4b15-4 (at 10.50.16.6@o2ib2) [479216.110516] Lustre: fir-MDT0002: haven't heard from client c4422e40-3cfa-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b3df39800, cur 1585033778 expire 1585033628 last 1585033551 [479216.130491] Lustre: Skipped 1 previous similar message [479235.379234] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [482341.195727] Lustre: fir-MDT0002: haven't heard from client 6e31591f-913a-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6b81a6ec00, cur 1585036903 expire 1585036753 last 1585036676 [482374.108628] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [483967.278313] Lustre: fir-MDT0002: Connection restored to 33701c5f-e220-4 (at 10.50.13.12@o2ib2) [484006.226224] Lustre: fir-MDT0002: haven't heard from client bd12e5c8-ece9-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba925c800, cur 1585038568 expire 1585038418 last 1585038341 [484228.447030] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [489237.357724] Lustre: fir-MDT0002: haven't heard from client 0fe03f2f-af46-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7f2d715c00, cur 1585043799 expire 1585043649 last 1585043572 [489259.138182] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [501277.677543] Lustre: fir-MDT0002: haven't heard from client ee56fbe2-040d-4 (at 10.49.25.17@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babbf2c9800, cur 1585055839 expire 1585055689 last 1585055612 [502758.701869] Lustre: fir-MDT0002: haven't heard from client d8c1b053-4ffb-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8addf6c800, cur 1585057320 expire 1585057170 last 1585057093 [502818.135642] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [510797.902020] Lustre: fir-MDT0002: haven't heard from client 89912c83-1f62-4 (at 10.50.14.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaf199800, cur 1585065359 expire 1585065209 last 1585065132 [512716.442803] Lustre: fir-MDT0002: Connection restored to 89912c83-1f62-4 (at 10.50.14.5@o2ib2) [512896.186514] Lustre: 20350:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585067450/real 1585067455] req@ffff8b9d81b8e300 x1661535681529408/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1585067457 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [512896.214633] Lustre: 20350:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [512896.224464] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [512896.238384] LustreError: Skipped 2 previous similar messages [512913.740414] LustreError: 11-0: fir-MDT0000-osp-MDT0002: operation ldlm_enqueue to node 10.0.10.52@o2ib7 failed: rc = -107 [512913.751460] LustreError: Skipped 16 previous similar messages [512913.757304] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [512913.773402] Lustre: Skipped 1 previous similar message [512946.387996] Lustre: fir-MDT0000-lwp-MDT0002: Connection to fir-MDT0000 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [512957.865460] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.22.35@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [512957.882916] LustreError: Skipped 2 previous similar messages [512958.734031] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.24.36@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [512958.751479] LustreError: Skipped 1 previous similar message [512960.118671] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.43@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [512960.136038] LustreError: Skipped 7 previous similar messages [512962.139029] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.22.9@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [512962.156396] LustreError: Skipped 28 previous similar messages [512966.141083] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.30.3@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [512966.158450] LustreError: Skipped 184 previous similar messages [512974.177663] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.3.8@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [512974.194947] LustreError: Skipped 362 previous similar messages [512990.285546] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.7.56@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [512990.302921] LustreError: Skipped 751 previous similar messages [513058.219933] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.22.35@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [513058.237393] LustreError: Skipped 40 previous similar messages [513089.558335] Lustre: fir-MDT0002: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [513114.071757] LNet: Service thread pid 41325 was inactive for 200.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [513114.088777] Pid: 41325, comm: mdt03_039 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [513114.099037] Call Trace: [513114.101590] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [513114.108271] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [513114.115048] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [513114.121809] [] osp_md_object_lock+0x162/0x2d0 [osp] [513114.128474] [] lod_object_lock+0xf4/0x780 [lod] [513114.134776] [] mdd_object_lock+0x3e/0xe0 [mdd] [513114.141005] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [513114.148358] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [513114.155200] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [513114.161513] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [513114.168097] [] mdt_reint_rec+0x83/0x210 [mdt] [513114.174251] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [513114.180910] [] mdt_reint+0x67/0x140 [mdt] [513114.186699] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [513114.193746] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [513114.201549] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [513114.207972] [] kthread+0xd1/0xe0 [513114.212979] [] ret_from_fork_nospec_begin+0xe/0x21 [513114.219553] [] 0xffffffffffffffff [513114.224663] LustreError: dumping log to /tmp/lustre-log.1585067675.41325 [513115.095787] LNet: Service thread pid 41323 was inactive for 200.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [513115.112807] Pid: 41323, comm: mdt01_042 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [513115.123067] Call Trace: [513115.125626] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [513115.132321] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [513115.139096] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [513115.145857] [] osp_md_object_lock+0x162/0x2d0 [osp] [513115.152527] [] lod_object_lock+0xf4/0x780 [lod] [513115.158840] [] mdd_object_lock+0x3e/0xe0 [mdd] [513115.165064] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [513115.172414] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [513115.179273] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [513115.185588] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [513115.192173] [] mdt_reint_rec+0x83/0x210 [mdt] [513115.198311] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [513115.204981] [] mdt_reint+0x67/0x140 [mdt] [513115.210775] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [513115.217825] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [513115.225629] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [513115.232056] [] kthread+0xd1/0xe0 [513115.237061] [] ret_from_fork_nospec_begin+0xe/0x21 [513115.243635] [] 0xffffffffffffffff [513115.248746] LustreError: dumping log to /tmp/lustre-log.1585067676.41323 [513115.607799] Pid: 20952, comm: mdt00_015 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [513115.618065] Call Trace: [513115.620622] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [513115.627327] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [513115.634119] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [513115.640881] [] osp_md_object_lock+0x162/0x2d0 [osp] [513115.647556] [] lod_object_lock+0xf4/0x780 [lod] [513115.653886] [] mdd_object_lock+0x3e/0xe0 [mdd] [513115.660113] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [513115.667495] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [513115.674326] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [513115.680641] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [513115.687215] [] mdt_reint_rec+0x83/0x210 [mdt] [513115.693374] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [513115.700034] [] mdt_reint+0x67/0x140 [mdt] [513115.705843] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [513115.712885] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [513115.720718] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [513115.727144] [] kthread+0xd1/0xe0 [513115.732167] [] ret_from_fork_nospec_begin+0xe/0x21 [513115.738730] [] 0xffffffffffffffff [513116.631825] LNet: Service thread pid 41468 was inactive for 200.31s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [513116.648846] LNet: Skipped 1 previous similar message [513116.653908] Pid: 41468, comm: mdt00_051 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [513116.664186] Call Trace: [513116.666740] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [513116.673436] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [513116.680216] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [513116.686998] [] osp_md_object_lock+0x162/0x2d0 [osp] [513116.693646] [] lod_object_lock+0xf4/0x780 [lod] [513116.699974] [] mdd_object_lock+0x3e/0xe0 [mdd] [513116.706191] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [513116.713572] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [513116.720400] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [513116.726730] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [513116.733293] [] mdt_reint_rec+0x83/0x210 [mdt] [513116.739448] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [513116.746103] [] mdt_reint+0x67/0x140 [mdt] [513116.751933] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [513116.758976] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [513116.766797] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [513116.773207] [] kthread+0xd1/0xe0 [513116.778225] [] ret_from_fork_nospec_begin+0xe/0x21 [513116.784780] [] 0xffffffffffffffff [513116.789891] LustreError: dumping log to /tmp/lustre-log.1585067677.41468 [513117.143833] Pid: 41551, comm: mdt02_061 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [513117.154097] Call Trace: [513117.156655] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [513117.163351] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [513117.170126] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [513117.176888] [] osp_md_object_lock+0x162/0x2d0 [osp] [513117.183550] [] lod_object_lock+0xf4/0x780 [lod] [513117.189863] [] mdd_object_lock+0x3e/0xe0 [mdd] [513117.196093] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [513117.203442] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [513117.210286] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [513117.216599] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [513117.223184] [] mdt_reint_rec+0x83/0x210 [mdt] [513117.229340] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [513117.236011] [] mdt_reint+0x67/0x140 [mdt] [513117.241803] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [513117.248847] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [513117.256651] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [513117.263078] [] kthread+0xd1/0xe0 [513117.268081] [] ret_from_fork_nospec_begin+0xe/0x21 [513117.274659] [] 0xffffffffffffffff [513117.279765] LustreError: dumping log to /tmp/lustre-log.1585067678.41551 [513117.287073] LNet: Service thread pid 20515 was inactive for 200.89s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [513127.384076] LNet: Service thread pid 41519 was inactive for 200.32s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [513127.397032] LustreError: dumping log to /tmp/lustre-log.1585067688.41519 [513130.456155] LustreError: dumping log to /tmp/lustre-log.1585067691.41523 [513132.504210] LustreError: dumping log to /tmp/lustre-log.1585067693.20966 [513136.088289] LustreError: dumping log to /tmp/lustre-log.1585067697.20863 [513150.936666] LNet: Service thread pid 20996 was inactive for 200.04s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [513150.949617] LNet: Skipped 3 previous similar messages [513150.954765] LustreError: dumping log to /tmp/lustre-log.1585067711.20996 [513172.185415] LustreError: 167-0: fir-MDT0000-lwp-MDT0002: This client was evicted by fir-MDT0000; in progress operations using this service will fail. [513172.208152] Lustre: fir-MDT0000-lwp-MDT0002: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [513176.025273] LustreError: dumping log to /tmp/lustre-log.1585067737.20914 [513181.260431] Lustre: fir-MDT0000-osp-MDT0002: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [513181.271232] LNet: Service thread pid 20952 completed after 265.95s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [513181.287499] LNet: Skipped 4 previous similar messages [513197.273978] Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0xae6d2c631bd4526c to 0x764984320c8ae934 [513197.286282] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [513258.265681] Lustre: fir-MDT0002: Connection restored to f7fe261e-a413-4 (at 10.49.28.2@o2ib1) [513460.968217] Lustre: fir-MDT0002: haven't heard from client 08262875-245e-4 (at 10.50.15.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb1713c00, cur 1585068022 expire 1585067872 last 1585067795 [514663.193643] Lustre: fir-MDT0002: Connection restored to 08262875-245e-4 (at 10.50.15.13@o2ib2) [514663.202346] Lustre: Skipped 2 previous similar messages [515891.618532] Lustre: fir-MDT0002: Connection restored to ce67c0db-bb4c-4 (at 10.50.15.5@o2ib2) [515936.220930] Lustre: fir-MDT0002: Connection restored to 6f426f61-5639-4 (at 10.50.13.10@o2ib2) [517754.072118] Lustre: fir-MDT0002: haven't heard from client c4e4a9fa-0416-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7966a5a800, cur 1585072315 expire 1585072165 last 1585072088 [517814.901525] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [518870.101668] Lustre: fir-MDT0002: haven't heard from client 82c5fb51-6200-4 (at 10.50.16.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babad2d8400, cur 1585073431 expire 1585073281 last 1585073204 [520612.535032] Lustre: fir-MDT0002: Connection restored to 82c5fb51-6200-4 (at 10.50.16.9@o2ib2) [522256.183289] Lustre: fir-MDT0002: haven't heard from client d4aafd71-b333-4 (at 10.50.14.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb2989400, cur 1585076817 expire 1585076667 last 1585076590 [523427.486393] Lustre: fir-MDT0002: Connection restored to d4aafd71-b333-4 (at 10.50.14.15@o2ib2) [528080.328421] Lustre: fir-MDT0002: haven't heard from client abed66c3-995c-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b4de24800, cur 1585082641 expire 1585082491 last 1585082414 [528109.977917] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [528327.335175] Lustre: fir-MDT0002: haven't heard from client 9f02a5e3-9696-4 (at 10.50.13.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb5799400, cur 1585082888 expire 1585082738 last 1585082661 [530257.405880] Lustre: fir-MDT0002: Connection restored to 9f02a5e3-9696-4 (at 10.50.13.13@o2ib2) [531690.417450] Lustre: fir-MDT0002: haven't heard from client 72866633-325f-4 (at 10.50.15.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba7fbc400, cur 1585086251 expire 1585086101 last 1585086024 [533400.363080] Lustre: fir-MDT0002: Connection restored to 72866633-325f-4 (at 10.50.15.9@o2ib2) [536381.533998] Lustre: fir-MDT0002: haven't heard from client e9949bf5-ceeb-4 (at 10.50.12.12@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babae6e9c00, cur 1585090942 expire 1585090792 last 1585090715 [537512.572096] Lustre: fir-MDT0002: Connection restored to e9949bf5-ceeb-4 (at 10.50.12.12@o2ib2) [542678.967217] Lustre: fir-MDT0002: Connection restored to da59e233-110f-4 (at 10.50.13.15@o2ib2) [557784.063327] Lustre: fir-MDT0002: haven't heard from client d95d81db-621a-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b96906c8000, cur 1585112344 expire 1585112194 last 1585112117 [557831.592866] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [558439.592623] Lustre: fir-MDT0002: Connection restored to 88c27327-e888-4 (at 10.49.27.21@o2ib1) [559231.798113] Lustre: fir-MDT0002: Client 87383dee-e650-4 (at 10.50.10.17@o2ib2) reconnecting [559231.806598] Lustre: fir-MDT0002: Connection restored to 87383dee-e650-4 (at 10.50.10.17@o2ib2) [567719.311395] Lustre: fir-MDT0002: haven't heard from client 9cf8861f-40bc-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b4dc7c000, cur 1585122279 expire 1585122129 last 1585122052 [567765.809835] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [570001.368273] Lustre: fir-MDT0002: haven't heard from client 140a8048-bada-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb9fb8c00, cur 1585124561 expire 1585124411 last 1585124334 [570079.713901] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [571587.409875] Lustre: fir-MDT0002: haven't heard from client 69ede8ab-5c22-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ab52abc00, cur 1585126147 expire 1585125997 last 1585125920 [571626.893541] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [578252.577087] Lustre: fir-MDT0002: haven't heard from client 53fb438a-2a45-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab4ce4f000, cur 1585132812 expire 1585132662 last 1585132585 [578308.099263] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [581596.664227] Lustre: fir-MDT0002: haven't heard from client 568515af-b94a-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b5956d000, cur 1585136156 expire 1585136006 last 1585135929 [581647.727327] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [584508.731214] Lustre: fir-MDT0002: haven't heard from client 51e206d5-3327-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b2f761400, cur 1585139068 expire 1585138918 last 1585138841 [584579.286082] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [588043.820962] Lustre: fir-MDT0002: haven't heard from client e685b13a-a4f9-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b98c3f18400, cur 1585142603 expire 1585142453 last 1585142376 [588093.581997] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [588792.840786] Lustre: fir-MDT0002: haven't heard from client f32d3637-39fe-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b5c5fd000, cur 1585143352 expire 1585143202 last 1585143125 [588834.335758] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [594305.982456] Lustre: fir-MDT0002: haven't heard from client dadffa87-4553-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baae0bf8800, cur 1585148865 expire 1585148715 last 1585148638 [594359.941209] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [596043.024294] Lustre: fir-MDT0002: haven't heard from client 2d5a47ed-4b4c-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ab5718c00, cur 1585150602 expire 1585150452 last 1585150375 [596092.006019] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [604456.548721] Lustre: fir-MDT0002: Connection restored to 51620f49-4677-4 (at 10.50.15.2@o2ib2) [615561.507829] Lustre: fir-MDT0002: haven't heard from client d7618323-c1e8-4 (at 10.50.14.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9aeeb33000, cur 1585170120 expire 1585169970 last 1585169893 [617467.489967] Lustre: fir-MDT0002: Connection restored to 430e4894-d38d-4 (at 10.50.14.11@o2ib2) [621797.636434] Lustre: fir-MDT0002: Client 02886e93-b578-4 (at 10.50.1.47@o2ib2) reconnecting [621797.644817] Lustre: fir-MDT0002: Connection restored to 02886e93-b578-4 (at 10.50.1.47@o2ib2) [633606.993036] Lustre: fir-MDT0002: haven't heard from client 68edeab9-3169-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb44c1c00, cur 1585188165 expire 1585188015 last 1585187938 [633650.423640] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [635457.998406] Lustre: fir-MDT0002: haven't heard from client 1015a07c-5c4c-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b90de6de400, cur 1585190016 expire 1585189866 last 1585189789 [635519.507415] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [637353.045750] Lustre: fir-MDT0002: haven't heard from client af897907-9595-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b58655800, cur 1585191911 expire 1585191761 last 1585191684 [637392.783274] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [660245.989739] Lustre: fir-MDT0002: Client bb56e686-3cef-4 (at 10.49.8.30@o2ib1) reconnecting [660245.998121] Lustre: fir-MDT0002: Connection restored to bb56e686-3cef-4 (at 10.49.8.30@o2ib1) [664449.634090] Lustre: fir-MDT0002: Client bb56e686-3cef-4 (at 10.49.8.30@o2ib1) reconnecting [664449.642467] Lustre: fir-MDT0002: Connection restored to bb56e686-3cef-4 (at 10.49.8.30@o2ib1) [664825.412591] Lustre: fir-MDT0002: Client bb56e686-3cef-4 (at 10.49.8.30@o2ib1) reconnecting [664825.420982] Lustre: fir-MDT0002: Connection restored to bb56e686-3cef-4 (at 10.49.8.30@o2ib1) [666248.400519] Lustre: fir-MDT0002: Client bb56e686-3cef-4 (at 10.49.8.30@o2ib1) reconnecting [666248.408892] Lustre: fir-MDT0002: Connection restored to bb56e686-3cef-4 (at 10.49.8.30@o2ib1) [682708.465857] Lustre: fir-MDT0002: Client bb56e686-3cef-4 (at 10.49.8.30@o2ib1) reconnecting [682708.474239] Lustre: fir-MDT0002: Connection restored to bb56e686-3cef-4 (at 10.49.8.30@o2ib1) [683334.686583] Lustre: fir-MDT0002: Client bb56e686-3cef-4 (at 10.49.8.30@o2ib1) reconnecting [683334.694962] Lustre: fir-MDT0002: Connection restored to bb56e686-3cef-4 (at 10.49.8.30@o2ib1) [686449.284352] Lustre: fir-MDT0002: haven't heard from client 8795bae7-a3eb-4 (at 10.50.12.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b83f0296c00, cur 1585241006 expire 1585240856 last 1585240779 [688395.872224] Lustre: fir-MDT0002: Connection restored to 38135a83-ec5c-4 (at 10.50.12.5@o2ib2) [693126.794330] Lustre: fir-MDT0002: Client ef345730-39be-4 (at 10.49.28.4@o2ib1) reconnecting [693126.802716] Lustre: fir-MDT0002: Connection restored to ef345730-39be-4 (at 10.49.28.4@o2ib1) [703974.511871] Lustre: fir-MDT0002: Connection restored to 6dcf0e67-1d32-4 (at 10.50.7.9@o2ib2) [703974.721628] Lustre: fir-MDT0002: haven't heard from client 6dcf0e67-1d32-4 (at 10.50.7.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babbf2c8000, cur 1585258531 expire 1585258381 last 1585258304 [705141.888711] Lustre: fir-MDT0002: Connection restored to 35cd2c2f-e02c-4 (at 10.50.12.13@o2ib2) [768382.841601] Lustre: fir-MDT0002: Connection restored to cb22fe63-8523-4 (at 10.49.19.4@o2ib1) [769219.380973] Lustre: fir-MDT0002: haven't heard from client 3ab8a3cd-4496-4 (at 10.49.19.3@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babad3d6800, cur 1585323774 expire 1585323624 last 1585323547 [770464.742236] Lustre: fir-MDT0002: Connection restored to cb22fe63-8523-4 (at 10.49.19.4@o2ib1) [770496.514749] Lustre: fir-MDT0002: Connection restored to 3ab8a3cd-4496-4 (at 10.49.19.3@o2ib1) [770665.060448] Lustre: fir-MDT0002: Connection restored to 6eb34c30-3804-4 (at 10.49.19.6@o2ib1) [771107.788330] Lustre: fir-MDT0002: Connection restored to 3ab8a3cd-4496-4 (at 10.49.19.3@o2ib1) [772255.685659] Lustre: fir-MDT0002: Connection restored to 6f68bfc7-8d0d-4 (at 10.49.19.1@o2ib1) [772255.694275] Lustre: Skipped 1 previous similar message [772283.446667] Lustre: fir-MDT0002: Connection restored to 565e39cd-4898-4 (at 10.49.19.8@o2ib1) [772319.519967] Lustre: fir-MDT0002: Connection restored to 0b2cd2d4-168b-4 (at 10.49.19.7@o2ib1) [772326.865090] Lustre: fir-MDT0002: Connection restored to 9939e999-d857-4 (at 10.49.19.5@o2ib1) [773270.134325] Lustre: fir-MDT0002: Connection restored to 6eb34c30-3804-4 (at 10.49.19.6@o2ib1) [773300.333532] Lustre: fir-MDT0002: Connection restored to 3ab8a3cd-4496-4 (at 10.49.19.3@o2ib1) [773300.342150] Lustre: Skipped 1 previous similar message [773319.752197] Lustre: fir-MDT0002: Connection restored to 6f68bfc7-8d0d-4 (at 10.49.19.1@o2ib1) [779384.130272] Lustre: fir-MDT0002: Client d736fd99-f618-4 (at 10.50.10.11@o2ib2) reconnecting [779384.138741] Lustre: fir-MDT0002: Connection restored to d736fd99-f618-4 (at 10.50.10.11@o2ib2) [783962.585402] Lustre: fir-MDT0002: Connection restored to 0b2cd2d4-168b-4 (at 10.49.19.7@o2ib1) [783969.695817] Lustre: fir-MDT0002: Connection restored to 9939e999-d857-4 (at 10.49.19.5@o2ib1) [791869.942899] Lustre: fir-MDT0002: haven't heard from client 661e22b8-e9f5-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7f2d7ce800, cur 1585346424 expire 1585346274 last 1585346197 [791869.962868] Lustre: Skipped 7 previous similar messages [791921.300595] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [794806.016001] Lustre: fir-MDT0002: haven't heard from client 144bfe18-3352-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab4fd30400, cur 1585349360 expire 1585349210 last 1585349133 [794852.397488] Lustre: fir-MDT0002: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [795574.917478] Lustre: fir-MDT0002: Connection restored to f3e067b1-f2a3-4 (at 10.49.19.2@o2ib1) [806257.602547] Lustre: fir-MDT0002: Client 26b36137-25da-4 (at 10.50.1.44@o2ib2) reconnecting [806257.610923] Lustre: fir-MDT0002: Connection restored to 26b36137-25da-4 (at 10.50.1.44@o2ib2) [810995.643986] Lustre: fir-MDT0002: Client d736fd99-f618-4 (at 10.50.10.11@o2ib2) reconnecting [810995.652455] Lustre: fir-MDT0002: Connection restored to d736fd99-f618-4 (at 10.50.10.11@o2ib2) [811393.000454] Lustre: fir-MDT0002: Client 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) reconnecting [811393.008837] Lustre: fir-MDT0002: Connection restored to 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) [811991.945417] Lustre: fir-MDT0002: Client 164e843a-d84a-4 (at 10.50.5.36@o2ib2) reconnecting [811991.953829] Lustre: fir-MDT0002: Connection restored to 164e843a-d84a-4 (at 10.50.5.36@o2ib2) [815444.832885] Lustre: fir-MDT0002: Client d736fd99-f618-4 (at 10.50.10.11@o2ib2) reconnecting [815444.841349] Lustre: fir-MDT0002: Connection restored to d736fd99-f618-4 (at 10.50.10.11@o2ib2) [815445.606498] Lustre: fir-MDT0002: Client 5b88c4a2-39df-4 (at 10.50.17.45@o2ib2) reconnecting [815445.614964] Lustre: fir-MDT0002: Connection restored to 5b88c4a2-39df-4 (at 10.50.17.45@o2ib2) [816748.133915] Lustre: fir-MDT0002: Client 164e843a-d84a-4 (at 10.50.5.36@o2ib2) reconnecting [816748.142314] Lustre: fir-MDT0002: Connection restored to 164e843a-d84a-4 (at 10.50.5.36@o2ib2) [816964.135063] Lustre: fir-MDT0002: Client d736fd99-f618-4 (at 10.50.10.11@o2ib2) reconnecting [816964.143527] Lustre: fir-MDT0002: Connection restored to d736fd99-f618-4 (at 10.50.10.11@o2ib2) [817105.691745] Lustre: fir-MDT0002: Client 164e843a-d84a-4 (at 10.50.5.36@o2ib2) reconnecting [817105.700127] Lustre: fir-MDT0002: Connection restored to 164e843a-d84a-4 (at 10.50.5.36@o2ib2) [817289.112840] Lustre: fir-MDT0002: Client 4e4443f1-69b2-4 (at 10.50.2.19@o2ib2) reconnecting [817289.121222] Lustre: fir-MDT0002: Connection restored to 4e4443f1-69b2-4 (at 10.50.2.19@o2ib2) [817952.434262] Lustre: fir-MDT0002: Client 4f300b23-846a-4 (at 10.50.1.1@o2ib2) reconnecting [817952.442558] Lustre: fir-MDT0002: Connection restored to 4f300b23-846a-4 (at 10.50.1.1@o2ib2) [818146.120743] Lustre: fir-MDT0002: Connection restored to 35cd2c2f-e02c-4 (at 10.50.12.13@o2ib2) [818639.425949] Lustre: fir-MDT0002: Client 8cb1fd67-2023-4 (at 10.50.1.49@o2ib2) reconnecting [818639.434330] Lustre: fir-MDT0002: Connection restored to 8cb1fd67-2023-4 (at 10.50.1.49@o2ib2) [818783.865468] Lustre: fir-MDT0002: Client 184a7f8c-61cb-4 (at 10.50.2.20@o2ib2) reconnecting [818783.873864] Lustre: fir-MDT0002: Connection restored to 184a7f8c-61cb-4 (at 10.50.2.20@o2ib2) [819015.259296] Lustre: fir-MDT0002: Client 184a7f8c-61cb-4 (at 10.50.2.20@o2ib2) reconnecting [819015.267681] Lustre: fir-MDT0002: Connection restored to 184a7f8c-61cb-4 (at 10.50.2.20@o2ib2) [838401.322894] Lustre: fir-MDT0002: Client 19361316-e119-4 (at 10.50.8.10@o2ib2) reconnecting [838401.331277] Lustre: fir-MDT0002: Connection restored to 19361316-e119-4 (at 10.50.8.10@o2ib2) [853187.462106] Lustre: fir-MDT0002: haven't heard from client a93d4107-2edc-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b9f4b3c00, cur 1585407740 expire 1585407590 last 1585407513 [853287.319410] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [870824.887233] Lustre: fir-MDT0002: haven't heard from client 85b52166-baef-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9284e24c00, cur 1585425377 expire 1585425227 last 1585425150 [870845.777464] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [880911.144850] Lustre: fir-MDT0002: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [880968.125513] Lustre: fir-MDT0002: haven't heard from client 780d3450-c327-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b55dbfc00, cur 1585435520 expire 1585435370 last 1585435293 [885349.227880] Lustre: fir-MDT0002: haven't heard from client add63273-c3b3-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b488d6800, cur 1585439901 expire 1585439751 last 1585439674 [885389.887545] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [886552.258552] Lustre: fir-MDT0002: haven't heard from client 9525333d-e5b4-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b74930a9000, cur 1585441104 expire 1585440954 last 1585440877 [886586.547409] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [905978.732745] Lustre: fir-MDT0002: haven't heard from client ecf003cb-b7be-4 (at 10.50.16.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb019a800, cur 1585460530 expire 1585460380 last 1585460303 [907733.185843] Lustre: fir-MDT0002: Connection restored to ecf003cb-b7be-4 (at 10.50.16.8@o2ib2) [935613.864282] Lustre: fir-MDT0002: Client 432d8224-80b5-4 (at 10.50.13.12@o2ib2) reconnecting [935613.872746] Lustre: fir-MDT0002: Connection restored to 33701c5f-e220-4 (at 10.50.13.12@o2ib2) [935615.542653] Lustre: fir-MDT0002: Client d736fd99-f618-4 (at 10.50.10.11@o2ib2) reconnecting [935615.551097] Lustre: Skipped 1 previous similar message [935615.556356] Lustre: fir-MDT0002: Connection restored to d736fd99-f618-4 (at 10.50.10.11@o2ib2) [935615.565061] Lustre: Skipped 1 previous similar message [935622.753598] Lustre: fir-MDT0002: Client 2f3a422b-dba8-4 (at 10.50.0.63@o2ib2) reconnecting [935622.761981] Lustre: fir-MDT0002: Connection restored to 2f3a422b-dba8-4 (at 10.50.0.63@o2ib2) [938060.518586] Lustre: fir-MDT0002: haven't heard from client bdc0155b-ddc0-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6b95909c00, cur 1585492611 expire 1585492461 last 1585492384 [938103.391385] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [943585.713530] Lustre: fir-MDT0002: haven't heard from client 144229db-845e-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baa0ea7ac00, cur 1585498136 expire 1585497986 last 1585497909 [943621.872884] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [944950.620187] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [946486.721748] Lustre: fir-MDT0002: haven't heard from client 6e79d69f-c682-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9b86fe6400, cur 1585501037 expire 1585500887 last 1585500810 [946521.752478] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [947057.738550] Lustre: fir-MDT0002: haven't heard from client dd1ee8c2-0420-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b87d52a5c00, cur 1585501608 expire 1585501458 last 1585501381 [947087.404399] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [953780.154233] Lustre: fir-MDT0002: Client cf0405d7-8447-4 (at 10.50.6.27@o2ib2) reconnecting [953780.162611] Lustre: fir-MDT0002: Connection restored to 4171bc70-1760-4 (at 10.50.6.27@o2ib2) [953782.344806] Lustre: fir-MDT0002: Client 432d8224-80b5-4 (at 10.50.13.12@o2ib2) reconnecting [953782.353252] Lustre: Skipped 2 previous similar messages [953782.358598] Lustre: fir-MDT0002: Connection restored to 33701c5f-e220-4 (at 10.50.13.12@o2ib2) [953782.367299] Lustre: Skipped 2 previous similar messages [953784.681750] Lustre: fir-MDT0002: Client 409757b7-db0e-4 (at 10.50.15.9@o2ib2) reconnecting [953784.690121] Lustre: Skipped 1 previous similar message [953784.695371] Lustre: fir-MDT0002: Connection restored to 72866633-325f-4 (at 10.50.15.9@o2ib2) [953784.703979] Lustre: Skipped 1 previous similar message [953788.691065] Lustre: fir-MDT0002: Client 202c81d9-fcd9-4 (at 10.50.4.41@o2ib2) reconnecting [953788.699436] Lustre: fir-MDT0002: Connection restored to 202c81d9-fcd9-4 (at 10.50.4.41@o2ib2) [954223.238588] LNetError: 20288:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [954223.248850] LNetError: 20288:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (105): c: 7, oc: 0, rc: 8 [954372.486592] Lustre: fir-MDT0002: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [954372.496936] Lustre: Skipped 1 previous similar message [955451.934379] Lustre: fir-MDT0002: haven't heard from client 0885c11b-bcdc-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8dacbc4c00, cur 1585510002 expire 1585509852 last 1585509775 [955451.954385] LustreError: 21603:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.9.37@o2ib2) failed to reply to blocking AST (req@ffff8b901ae03a80 x1661546863905536 status 0 rc -5), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b97e368ba80/0x2f21cf22cd6f9921 lrc: 4/0,0 mode: PR/PR res: [0x2c00390ed:0x14265:0x0].0x0 bits 0x13/0x0 rrc: 579 type: IBT flags: 0x60200400000020 nid: 10.50.9.37@o2ib2 remote: 0x2813fe829cb99b4b expref: 4411 pid: 21565 timeout: 955577 lvb_type: 0 [955451.997486] LustreError: 138-a: fir-MDT0002: A client on nid 10.50.9.37@o2ib2 was evicted due to a lock blocking callback time out: rc -5 [955487.990883] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [964712.567416] Lustre: 41534:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585519255/real 1585519255] req@ffff8b8cb5204380 x1661546994105408/t0(0) o104->fir-MDT0002@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585519262 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [964719.594588] Lustre: 41534:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585519262/real 1585519262] req@ffff8b8cb5204380 x1661546994105408/t0(0) o104->fir-MDT0002@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585519269 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [964726.621777] Lustre: 41534:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585519269/real 1585519269] req@ffff8b8cb5204380 x1661546994105408/t0(0) o104->fir-MDT0002@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585519276 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [964733.648959] Lustre: 41534:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585519276/real 1585519276] req@ffff8b8cb5204380 x1661546994105408/t0(0) o104->fir-MDT0002@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585519283 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [964747.676345] Lustre: 41534:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585519290/real 1585519290] req@ffff8b8cb5204380 x1661546994105408/t0(0) o104->fir-MDT0002@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585519297 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [964747.703682] Lustre: 41534:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [964768.713893] Lustre: 41534:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585519311/real 1585519311] req@ffff8b8cb5204380 x1661546994105408/t0(0) o104->fir-MDT0002@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585519318 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [964768.741235] Lustre: 41534:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [964793.159929] Lustre: fir-MDT0002: haven't heard from client 8c7a79ff-f053-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b488dfc00, cur 1585519343 expire 1585519193 last 1585519116 [964883.707406] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [987708.730885] Lustre: fir-MDT0002: haven't heard from client 141984c0-0bb7-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b5973a000, cur 1585542258 expire 1585542108 last 1585542031 [987751.778461] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [989434.773983] Lustre: fir-MDT0002: haven't heard from client 3ca91764-a501-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b488ecc00, cur 1585543984 expire 1585543834 last 1585543757 [989481.828310] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [997912.980147] Lustre: fir-MDT0002: haven't heard from client 76521fe8-386d-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ab5799c00, cur 1585552462 expire 1585552312 last 1585552235 [997960.525861] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1029144.331977] Lustre: fir-MDT0002: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [1029212.738987] Lustre: fir-MDT0002: haven't heard from client fdd6ec41-5cf5-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b90df6f4c00, cur 1585583761 expire 1585583611 last 1585583534 [1033737.622945] Lustre: fir-MDT0002: Client 1521e60c-0011-4 (at 10.50.4.51@o2ib2) reconnecting [1033737.631385] Lustre: Skipped 1 previous similar message [1033737.636722] Lustre: fir-MDT0002: Connection restored to 1521e60c-0011-4 (at 10.50.4.51@o2ib2) [1056407.413924] Lustre: fir-MDT0002: haven't heard from client e47349d1-c447-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baa84fe7000, cur 1585610955 expire 1585610805 last 1585610728 [1058997.866340] Lustre: fir-MDT0002: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1058997.875060] Lustre: Skipped 1 previous similar message [1059111.541730] Lustre: fir-MDT0002: Client 8156a778-c974-4 (at 10.49.7.12@o2ib1) reconnecting [1059111.550175] Lustre: Skipped 1 previous similar message [1059111.555520] Lustre: fir-MDT0002: Connection restored to 8156a778-c974-4 (at 10.49.7.12@o2ib1) [1059112.769992] Lustre: fir-MDT0002: Client 8e2770c2-1687-4 (at 10.49.20.4@o2ib1) reconnecting [1059112.778468] Lustre: fir-MDT0002: Connection restored to 8e2770c2-1687-4 (at 10.49.20.4@o2ib1) [1059113.836297] Lustre: fir-MDT0002: Client 2134b4ec-8d65-4 (at 10.49.18.14@o2ib1) reconnecting [1059113.844824] Lustre: Skipped 4 previous similar messages [1059114.827910] Lustre: fir-MDT0002: Connection restored to 2b2221c8-ab32-4 (at 10.49.18.15@o2ib1) [1059114.836723] Lustre: Skipped 14 previous similar messages [1059115.954768] Lustre: fir-MDT0002: Client 4cb83747-634b-4 (at 10.49.8.26@o2ib1) reconnecting [1059115.963207] Lustre: Skipped 25 previous similar messages [1059118.857910] Lustre: fir-MDT0002: Connection restored to b7c30a0f-2fff-4 (at 10.49.8.23@o2ib1) [1059118.866614] Lustre: Skipped 45 previous similar messages [1059119.958093] Lustre: fir-MDT0002: Client 71f7d975-5539-4 (at 10.49.25.16@o2ib1) reconnecting [1059119.966618] Lustre: Skipped 48 previous similar messages [1059126.887356] Lustre: fir-MDT0002: Connection restored to 9823016a-d5f7-4 (at 10.49.0.63@o2ib1) [1059126.896055] Lustre: Skipped 250 previous similar messages [1059127.962924] Lustre: fir-MDT0002: Client 42700bac-98c8-4 (at 10.49.17.17@o2ib1) reconnecting [1059127.971456] Lustre: Skipped 232 previous similar messages [1059143.504554] Lustre: fir-MDT0002: Connection restored to 2a2adc1f-d7d0-4 (at 10.49.28.12@o2ib1) [1059143.513345] Lustre: Skipped 6 previous similar messages [1059146.564828] Lustre: fir-MDT0002: Client 4e51c703-4a03-4 (at 10.49.18.26@o2ib1) reconnecting [1059146.573360] Lustre: Skipped 6 previous similar messages [1059176.055123] Lustre: fir-MDT0002: Connection restored to a7cfebd8-0c3b-4 (at 10.49.27.17@o2ib1) [1059176.063911] Lustre: Skipped 93 previous similar messages [1059178.598889] Lustre: fir-MDT0002: Client 05195c7f-e5e4-4 (at 10.49.20.27@o2ib1) reconnecting [1059178.607417] Lustre: Skipped 95 previous similar messages [1059204.975282] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1059204.992734] LustreError: Skipped 1378 previous similar messages [1059229.850922] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1059243.860603] Lustre: fir-MDT0002: Client 55a7e825-64fb-4 (at 10.49.28.12@o2ib1) reconnecting [1059243.869128] Lustre: Skipped 321 previous similar messages [1059243.874738] Lustre: fir-MDT0002: Connection restored to 2a2adc1f-d7d0-4 (at 10.49.28.12@o2ib1) [1059243.883522] Lustre: Skipped 325 previous similar messages [1059363.998471] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1059398.808169] Lustre: fir-MDT0002: Client 0271c837-bb5b-4 (at 10.49.27.13@o2ib1) reconnecting [1059398.816696] Lustre: Skipped 96 previous similar messages [1059398.822231] Lustre: fir-MDT0002: Connection restored to 0271c837-bb5b-4 (at 10.49.27.13@o2ib1) [1059398.831029] Lustre: Skipped 96 previous similar messages [1059445.489904] Lustre: 74060:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8b8c6999d850 x1660842365181376/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059445.997763] Lustre: 57891:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8b994d61a400 x1660842367290752/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059446.020208] Lustre: 57891:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 654 previous similar messages [1059446.503003] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059446.512492] Lustre: 74057:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=387 reqQ=27549 recA=0, svcEst=36, delay=202 [1059446.523455] Lustre: 74057:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b9b7b30c380 x1660842364385984/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:743/0 lens 328/0 e 0 to 0 dl 1585613993 ref 2 fl New:/2/ffffffff rc 0/-1 [1059447.008371] Lustre: 46485:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s req@ffff8b8d953a0050 x1660842378780032/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059447.030599] Lustre: 46485:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 1182 previous similar messages [1059447.056059] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059447.065572] Lustre: Skipped 2 previous similar messages [1059447.065892] Lustre: 21006:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=94 reqQ=27629 recA=0, svcEst=36, delay=9 [1059447.065895] Lustre: 21006:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 2 previous similar messages [1059447.065903] Lustre: 21006:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b974e63bf00 x1660842362105024/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:743/0 lens 328/0 e 0 to 0 dl 1585613993 ref 2 fl New:/2/ffffffff rc 0/-1 [1059447.065905] Lustre: 21006:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 779 previous similar messages [1059448.069110] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059448.074900] Lustre: 74056:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=56 reqQ=27637 recA=0, svcEst=36, delay=5 [1059448.074906] Lustre: 74056:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 22 previous similar messages [1059448.074943] Lustre: 74056:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-2s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b978df18d80 x1660842352009280/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:743/0 lens 328/0 e 0 to 0 dl 1585613993 ref 2 fl New:/2/ffffffff rc 0/-1 [1059448.074948] Lustre: 74056:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 1589 previous similar messages [1059448.138642] Lustre: Skipped 21 previous similar messages [1059449.013345] Lustre: 96639:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 9s req@ffff8b9a89a51200 x1660842366520960/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059449.035565] Lustre: 96639:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 2207 previous similar messages [1059450.161506] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059450.171026] Lustre: Skipped 17 previous similar messages [1059450.177119] Lustre: 20492:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=299 reqQ=29185 recA=0, svcEst=36, delay=230 [1059450.188432] Lustre: 20492:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 20 previous similar messages [1059450.198199] Lustre: 20492:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-4s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b92f82ada00 x1660842337181504/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:743/0 lens 328/0 e 0 to 0 dl 1585613993 ref 2 fl New:/2/ffffffff rc 0/-1 [1059450.227786] Lustre: 20492:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 2722 previous similar messages [1059453.021335] Lustre: 74061:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 12s req@ffff8b994d738000 x1660842348814208/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059453.043765] Lustre: 74061:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 3802 previous similar messages [1059454.289102] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059454.298579] Lustre: Skipped 24 previous similar messages [1059454.304072] Lustre: 74058:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=305 reqQ=33466 recA=0, svcEst=36, delay=362 [1059454.315032] Lustre: 74058:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 24 previous similar messages [1059454.324780] Lustre: 74058:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-7s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b9a484bd100 x1660842343720448/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:744/0 lens 328/0 e 0 to 0 dl 1585613994 ref 2 fl New:/2/ffffffff rc 0/-1 [1059454.354022] Lustre: 74058:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 2234 previous similar messages [1059461.031498] Lustre: 47159:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 20s req@ffff8b7bb5e8ba80 x1660842373824128/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059461.053808] Lustre: 47159:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 3716 previous similar messages [1059463.042105] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059463.051592] Lustre: Skipped 56 previous similar messages [1059463.057088] Lustre: 21924:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=113 reqQ=4225 recA=0, svcEst=36, delay=1040 [1059463.068038] Lustre: 21924:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 56 previous similar messages [1059463.077779] Lustre: 21924:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8ba965ca4c80 x1660842345185792/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:4/0 lens 328/0 e 0 to 0 dl 1585614009 ref 2 fl New:/2/ffffffff rc 0/-1 [1059463.106841] Lustre: 21924:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 4960 previous similar messages [1059475.434561] LustreError: 21107:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.50.10.50@o2ib2: deadline 6:7s ago req@ffff8ba963595100 x1659416927471872/t0(0) o103->e356597a-d009-4@10.50.10.50@o2ib2:10/0 lens 328/0 e 0 to 0 dl 1585614015 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 [1059475.464939] Lustre: 21107:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:7s); client may timeout. req@ffff8ba963595100 x1659416927471872/t0(0) o103->e356597a-d009-4@10.50.10.50@o2ib2:10/0 lens 328/0 e 0 to 0 dl 1585614015 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 [1059477.036132] Lustre: 46751:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 13s req@ffff8b7d50a93050 x1660842337727360/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059477.058438] Lustre: 46751:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 13548 previous similar messages [1059479.223972] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059479.234267] Lustre: Skipped 110 previous similar messages [1059479.239853] Lustre: 74063:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=47 reqQ=9240 recA=0, svcEst=68, delay=712 [1059479.250628] Lustre: 74063:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 110 previous similar messages [1059479.260456] Lustre: 74063:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8ba8cfbabf00 x1660842345616704/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:10/0 lens 328/0 e 0 to 0 dl 1585614015 ref 2 fl New:/2/ffffffff rc 0/-1 [1059479.289690] Lustre: 74063:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 10459 previous similar messages [1059493.799112] Lustre: 46751:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (58/39), not sending early reply req@ffff8b8244355050 x1659488424780992/t0(0) o103->324a68c5-1e46-4@10.50.17.47@o2ib2:39/0 lens 328/0 e 0 to 0 dl 1585614099 ref 2 fl New:/2/ffffffff rc 0/-1 [1059500.911090] Lustre: 74066:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (42/38), not sending early reply req@ffff8baa04f86300 x1659413132608896/t0(0) o103->5b88c4a2-39df-4@10.50.17.45@o2ib2:48/0 lens 328/0 e 0 to 0 dl 1585614090 ref 2 fl New:/0/ffffffff rc 0/-1 [1059509.039669] Lustre: 46751:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 30s req@ffff8b819a316c00 x1660842344426944/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059509.061976] Lustre: 46751:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 13138 previous similar messages [1059511.522577] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059511.532061] Lustre: Skipped 154 previous similar messages [1059511.537640] Lustre: 49048:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=3057 reqQ=13599 recA=1, svcEst=99, delay=477 [1059511.548512] Lustre: 46750:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-26s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b8996eda400 x1660842344699392/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:28/0 lens 328/0 e 0 to 0 dl 1585614033 ref 2 fl New:/2/ffffffff rc 0/-1 [1059511.548515] Lustre: 46750:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 13065 previous similar messages [1059511.588381] Lustre: 49048:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 161 previous similar messages [1059524.960343] LustreError: 20489:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.50.10.50@o2ib2: deadline 6:26s ago req@ffff8b8b8e620850 x1659416927471872/t0(0) o103->e356597a-d009-4@10.50.10.50@o2ib2:41/0 lens 328/0 e 0 to 0 dl 1585614046 ref 2 fl Interpret:H/2/ffffffff rc 0/-1 [1059524.990829] Lustre: 20489:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:26s); client may timeout. req@ffff8b8b8e620850 x1659416927471872/t0(0) o103->e356597a-d009-4@10.50.10.50@o2ib2:41/0 lens 328/0 e 0 to 0 dl 1585614046 ref 1 fl Interpret:H/2/ffffffff rc 0/-1 [1059525.121092] Lustre: 41473:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1585614061/real 0] req@ffff8b6bbcbe3180 x1661549495998144/t0(0) o104->fir-MDT0002@10.49.0.63@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1585614072 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [1059525.147749] Lustre: 41473:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [1059627.498467] Lustre: 74056:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8b967da0b180 x1660842348722880/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059627.520692] Lustre: 74056:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 11662 previous similar messages [1059628.620370] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059628.620380] Lustre: 46749:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b9b11eead00 x1660842368540096/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:170/0 lens 328/0 e 0 to 0 dl 1585614175 ref 2 fl New:/2/ffffffff rc 0/-1 [1059628.620383] Lustre: 46749:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 8125 previous similar messages [1059628.654101] Lustre: 49039:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=102 reqQ=39117 recA=0, svcEst=111, delay=312 [1059628.654104] Lustre: 49039:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 96 previous similar messages [1059628.690269] Lustre: Skipped 106 previous similar messages [1059653.549158] Lustre: 46738:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:18s); client may timeout. req@ffff8b922bb05e80 x1660842342966400/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:178/0 lens 328/0 e 0 to 0 dl 1585614183 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [1059653.571373] LustreError: 21061:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.50.8.1@o2ib2: deadline 6:1s ago req@ffff8b7bb5e88d80 x1660863721735936/t0(0) o103->202dfe5b-bc78-4@10.50.8.1@o2ib2:195/0 lens 328/0 e 0 to 0 dl 1585614200 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [1059653.571376] LustreError: 21124:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.50.8.1@o2ib2: deadline 6:1s ago req@ffff8b7bb5e8ba80 x1660863720965056/t0(0) o103->202dfe5b-bc78-4@10.50.8.1@o2ib2:195/0 lens 328/0 e 0 to 0 dl 1585614200 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [1059653.571378] LustreError: 21061:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 47 previous similar messages [1059653.571379] LustreError: 21124:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [1059653.658066] Lustre: 46738:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 69572 previous similar messages [1059655.286211] Lustre: fir-MDT0002: Client 202dfe5b-bc78-4 (at 10.50.8.1@o2ib2) reconnecting [1059655.294584] Lustre: Skipped 380 previous similar messages [1059655.300190] Lustre: fir-MDT0002: Connection restored to 202dfe5b-bc78-4 (at 10.50.8.1@o2ib2) [1059655.308804] Lustre: Skipped 380 previous similar messages [1059676.500025] Lustre: fir-MDT0002: haven't heard from client c0c00923-1be8-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b803a372400, cur 1585614224 expire 1585614074 last 1585613997 [1059755.503086] Lustre: 49041:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 15s req@ffff8b8eb55fe780 x1660842343048576/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1059755.525397] Lustre: 49041:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 56822 previous similar messages [1059758.013850] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1059758.023324] Lustre: Skipped 471 previous similar messages [1059758.028904] Lustre: 20493:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=17 reqQ=65539 recA=0, svcEst=111, delay=2616 [1059758.039940] Lustre: 20493:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 475 previous similar messages [1059758.050152] Lustre: 20493:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-13s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b9b15153a80 x1660842336187840/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:287/0 lens 328/0 e 0 to 0 dl 1585614292 ref 2 fl New:/2/ffffffff rc 0/-1 [1059758.079484] Lustre: 20493:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 33863 previous similar messages [1059758.196009] Lustre: 46742:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:13s); client may timeout. req@ffff8b8eb67b9680 x1660842346018048/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:287/0 lens 328/0 e 0 to 0 dl 1585614292 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [1059758.223270] Lustre: 46742:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 5134 previous similar messages [1059865.581300] Lustre: 20496:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:2s); client may timeout. req@ffff8ba901323600 x1660842373792064/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:406/0 lens 328/0 e 0 to 0 dl 1585614411 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [1059865.608534] Lustre: 20496:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2687 previous similar messages [1059892.614955] Lustre: 21662:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:20s); client may timeout. req@ffff8b8c15bcb600 x1660842348826880/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:415/0 lens 328/0 e 0 to 0 dl 1585614420 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [1059892.642209] Lustre: 21662:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 11211 previous similar messages [1059892.652798] LustreError: 74066:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.49.8.24@o2ib1: deadline 6:6s ago req@ffff8baba9e8d850 x1659122707346432/t0(0) o103->9bb420b9-4b7e-4@10.49.8.24@o2ib1:429/0 lens 328/0 e 0 to 0 dl 1585614434 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [1059892.652800] LustreError: 43231:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.49.8.24@o2ib1: deadline 6:6s ago req@ffff8ba8d23e4800 x1659122707346048/t0(0) o103->9bb420b9-4b7e-4@10.49.8.24@o2ib1:429/0 lens 328/0 e 0 to 0 dl 1585614434 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [1059892.652801] LustreError: 74066:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 82 previous similar messages [1059892.652803] LustreError: 43231:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 82 previous similar messages [1059982.426277] LustreError: 49045:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.49.27.31@o2ib1: deadline 6:1s ago req@ffff8b7d0ded1b00 x1659514857309056/t0(0) o103->232937e8-bc43-4@10.49.27.31@o2ib1:523/0 lens 328/0 e 0 to 0 dl 1585614528 ref 2 fl Interpret:H/0/ffffffff rc 0/-1 [1059982.456737] LustreError: 49045:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 16 previous similar messages [1059982.467521] Lustre: 49045:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:1s); client may timeout. req@ffff8b7d0ded1b00 x1659514857309056/t0(0) o103->232937e8-bc43-4@10.49.27.31@o2ib1:523/0 lens 328/0 e 0 to 0 dl 1585614528 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 [1059982.494855] Lustre: 49045:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1142 previous similar messages [1059990.225434] Lustre: 21585:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585614530/real 1585614530] req@ffff8b953ea73600 x1661549501080640/t0(0) o104->fir-MDT0002@10.49.0.61@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1585614537 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1060097.507183] Lustre: 69813:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8b957079c380 x1660842343132928/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [1060097.529405] Lustre: 69813:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 77729 previous similar messages [1060098.610329] Lustre: 21063:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b8bafdbf050 x1660842338880512/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:640/0 lens 328/0 e 0 to 0 dl 1585614645 ref 2 fl New:/2/ffffffff rc 0/-1 [1060098.639576] Lustre: 21063:0:(service.c:1322:ptlrpc_at_send_early_reply()) Skipped 33138 previous similar messages [1060099.619866] Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). [1060099.619932] Lustre: 20981:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=35 reqQ=25087 recA=0, svcEst=62, delay=1347 [1060099.619935] Lustre: 20981:0:(service.c:1541:ptlrpc_at_check_timed()) Skipped 557 previous similar messages [1060099.650129] Lustre: Skipped 560 previous similar messages [1060099.731937] Lustre: 21107:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:2s); client may timeout. req@ffff8b9980a37980 x1660842338972544/t0(0) o103->0dc74af1-84ad-4@10.49.0.61@o2ib1:640/0 lens 328/0 e 0 to 0 dl 1585614645 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [1060099.739186] LustreError: 46734:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.50.8.25@o2ib2: deadline 6:1s ago req@ffff8b8d9ff49f80 x1659147546190912/t0(0) o103->83cddc96-7e25-4@10.50.8.25@o2ib2:641/0 lens 328/0 e 0 to 0 dl 1585614646 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [1060099.739190] LustreError: 46734:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 3 previous similar messages [1060099.799991] Lustre: 21107:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 8966 previous similar messages [1060116.893629] perf: interrupt took too long (4141 > 3981), lowering kernel.perf_event_max_sample_rate to 48000 [1060122.854802] LustreError: 20500:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.49.0.61@o2ib1 ns: mdt-fir-MDT0002_UUID lock: ffff8b7da2643a80/0x2f21cf23d792af6a lrc: 3/0,0 mode: PR/PR res: [0x2c0037a06:0x4676:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.49.0.61@o2ib1 remote: 0x727a5971c741b20b expref: 2155765 pid: 41528 timeout: 1060096 lvb_type: 0 [1060206.899101] Lustre: fir-MDT0002: Connection restored to 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) [1060206.907818] Lustre: Skipped 26 previous similar messages [1060272.858507] LustreError: 20500:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.49.0.61@o2ib1 ns: mdt-fir-MDT0002_UUID lock: ffff8b9b5abfee40/0x2f21cf23d7cee34f lrc: 3/0,0 mode: PR/PR res: [0x2c0037a70:0x66e2:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.49.0.61@o2ib1 remote: 0x727a5971c741f51e expref: 1360154 pid: 20920 timeout: 1060246 lvb_type: 0 [1060373.707977] LNet: Service thread pid 41546 was inactive for 200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1060373.725092] LNet: Skipped 1 previous similar message [1060373.730244] Pid: 41546, comm: mdt00_070 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1060373.740594] Call Trace: [1060373.743236] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [1060373.750370] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [1060373.757757] [] mdt_object_local_lock+0x438/0xb20 [mdt] [1060373.764774] [] mdt_object_lock_internal+0x70/0x360 [mdt] [1060373.771977] [] mdt_object_lock+0x20/0x30 [mdt] [1060373.778292] [] mdt_reint_open+0x106a/0x3240 [mdt] [1060373.784871] [] mdt_reint_rec+0x83/0x210 [mdt] [1060373.791102] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1060373.797846] [] mdt_intent_open+0x82/0x3a0 [mdt] [1060373.804250] [] mdt_intent_policy+0x435/0xd80 [mdt] [1060373.810906] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1060373.817840] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1060373.825129] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1060373.831495] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1060373.838617] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1060373.846509] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1060373.853023] [] kthread+0xd1/0xe0 [1060373.858116] [] ret_from_fork_nospec_begin+0xe/0x21 [1060373.864772] [] 0xffffffffffffffff [1060373.869978] LustreError: dumping log to /tmp/lustre-log.1585614921.41546 [1060473.401440] LustreError: 41546:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1585614720, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b73ad3e9440/0x2f21cf23d7ee16bf lrc: 3/0,1 mode: --/CW res: [0x2c0037a70:0x66e2:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 41546 timeout: 0 lvb_type: 0 [1060572.228631] LNet: Service thread pid 41546 completed after 398.82s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1060572.244960] LNet: Skipped 5 previous similar messages [1061889.753539] Lustre: fir-MDT0002: Client bb56e686-3cef-4 (at 10.49.8.30@o2ib1) reconnecting [1061889.761979] Lustre: Skipped 25 previous similar messages [1061889.767496] Lustre: fir-MDT0002: Connection restored to bb56e686-3cef-4 (at 10.49.8.30@o2ib1) [1079282.583375] Lustre: fir-MDT0002: Client 676dbe45-fcbb-4 (at 10.50.2.26@o2ib2) reconnecting [1079282.591838] Lustre: fir-MDT0002: Connection restored to 676dbe45-fcbb-4 (at 10.50.2.26@o2ib2) [1103776.605394] Lustre: fir-MDT0002: haven't heard from client e15458cc-5a0f-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7872f40c00, cur 1585658323 expire 1585658173 last 1585658096 [1103821.102308] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1103821.111013] Lustre: Skipped 2 previous similar messages [1108193.654691] Lustre: fir-MDT0002: Client c742d3fe-d253-4 (at 10.49.8.33@o2ib1) reconnecting [1108193.663133] Lustre: Skipped 2 previous similar messages [1108193.668563] Lustre: fir-MDT0002: Connection restored to c742d3fe-d253-4 (at 10.49.8.33@o2ib1) [1116793.927951] Lustre: fir-MDT0002: haven't heard from client 540b2ede-974e-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b3625a400, cur 1585671340 expire 1585671190 last 1585671113 [1116881.810334] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1127547.187316] Lustre: fir-MDT0002: haven't heard from client 71c512fc-f1b5-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b8cc95800, cur 1585682093 expire 1585681943 last 1585681866 [1127596.731860] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1149964.745861] Lustre: fir-MDT0002: haven't heard from client 21c8c585-5f1e-4 (at 10.50.8.18@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab9d741c00, cur 1585704510 expire 1585704360 last 1585704283 [1156059.304617] Lustre: fir-MDT0002: Client d3db911c-747e-4 (at 10.49.0.62@o2ib1) reconnecting [1156059.313096] Lustre: fir-MDT0002: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [1156067.938483] Lustre: fir-MDT0002: Client 4683044c-87cf-4 (at 10.49.27.16@o2ib1) reconnecting [1156067.947032] Lustre: fir-MDT0002: Connection restored to 4683044c-87cf-4 (at 10.49.27.16@o2ib1) [1179772.499758] Lustre: fir-MDT0002: haven't heard from client 51a2b347-1573-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba3fe7ebc00, cur 1585734317 expire 1585734167 last 1585734090 [1179819.697957] Lustre: fir-MDT0002: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [1206787.168974] Lustre: fir-MDT0002: haven't heard from client 39fa7b7d-0126-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b9f4b4800, cur 1585761331 expire 1585761181 last 1585761104 [1206832.871362] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1212843.894900] Lustre: fir-MDT0002: Client 8cb1fd67-2023-4 (at 10.50.1.49@o2ib2) reconnecting [1212843.903377] Lustre: fir-MDT0002: Connection restored to 8cb1fd67-2023-4 (at 10.50.1.49@o2ib2) [1212845.941467] Lustre: fir-MDT0002: Client aefc1bcc-c816-4 (at 10.50.2.25@o2ib2) reconnecting [1212845.949939] Lustre: fir-MDT0002: Connection restored to aefc1bcc-c816-4 (at 10.50.2.25@o2ib2) [1212849.390899] Lustre: fir-MDT0002: Client 00f921f1-6780-4 (at 10.50.10.55@o2ib2) reconnecting [1212849.399451] Lustre: fir-MDT0002: Connection restored to 00f921f1-6780-4 (at 10.50.10.55@o2ib2) [1212855.194920] Lustre: fir-MDT0002: Client 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) reconnecting [1212855.203363] Lustre: Skipped 1 previous similar message [1212855.208708] Lustre: fir-MDT0002: Connection restored to 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) [1212855.217433] Lustre: Skipped 1 previous similar message [1212859.958036] Lustre: fir-MDT0002: Client e361cff3-f0f6-4 (at 10.50.9.49@o2ib2) reconnecting [1212859.966501] Lustre: fir-MDT0002: Connection restored to e361cff3-f0f6-4 (at 10.50.9.49@o2ib2) [1212880.447112] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.0.71@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1212897.222588] Lustre: fir-MDT0002: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [1212897.231036] Lustre: Skipped 2 previous similar messages [1212897.236493] Lustre: fir-MDT0002: Connection restored to 64442903-03be-4 (at 10.50.0.61@o2ib2) [1212897.245197] Lustre: Skipped 2 previous similar messages [1212955.712043] Lustre: fir-MDT0002: Client 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) reconnecting [1212955.720507] Lustre: fir-MDT0002: Connection restored to 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) [1213025.719244] Lustre: fir-MDT0002: Client fe8a70df-a085-4 (at 10.50.10.2@o2ib2) reconnecting [1213025.727714] Lustre: fir-MDT0002: Connection restored to fe8a70df-a085-4 (at 10.50.10.2@o2ib2) [1215238.820570] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1215238.829363] Lustre: Skipped 4 previous similar messages [1215285.382579] Lustre: fir-MDT0002: haven't heard from client b449688d-0d11-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb173a000, cur 1585769829 expire 1585769679 last 1585769602 [1215719.737326] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1215768.393569] Lustre: fir-MDT0002: haven't heard from client f23e2d13-b3ae-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b90973dd800, cur 1585770312 expire 1585770162 last 1585770085 [1216324.420334] Lustre: fir-MDT0002: haven't heard from client f0da1f04-da4c-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b90012c1c00, cur 1585770868 expire 1585770718 last 1585770641 [1216327.833540] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1231484.438155] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1231536.793047] Lustre: fir-MDT0002: haven't heard from client 1b6de69e-1c49-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b944ae7b800, cur 1585786080 expire 1585785930 last 1585785853 [1261582.627622] Lustre: fir-MDT0002: haven't heard from client 479a23d1-e582-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b08fdb800, cur 1585816125 expire 1585815975 last 1585815898 [1261649.373570] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1274640.188367] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1274689.891666] Lustre: fir-MDT0002: haven't heard from client 7fa477f6-65a5-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba2696f4000, cur 1585829232 expire 1585829082 last 1585829005 [1275225.493216] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1275269.907743] Lustre: fir-MDT0002: haven't heard from client b34989b7-841a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9748ec5400, cur 1585829812 expire 1585829662 last 1585829585 [1275478.913171] Lustre: fir-MDT0002: haven't heard from client aaf569c5-b770-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baaedb07c00, cur 1585830021 expire 1585829871 last 1585829794 [1275677.923666] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1275928.380887] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1276006.938868] Lustre: fir-MDT0002: haven't heard from client 0bcab18e-ca9e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8f98e8cc00, cur 1585830549 expire 1585830399 last 1585830322 [1276322.326609] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1276357.936863] Lustre: fir-MDT0002: haven't heard from client 23290f18-a3b6-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b986ef60000, cur 1585830900 expire 1585830750 last 1585830673 [1276827.004008] Lustre: fir-MDT0002: haven't heard from client 4c28cfb2-f28b-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b97c3e7d000, cur 1585831369 expire 1585831219 last 1585831142 [1279128.668890] Lustre: fir-MDT0002: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [1279188.007859] Lustre: fir-MDT0002: haven't heard from client df593509-e01a-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab283de400, cur 1585833730 expire 1585833580 last 1585833503 [1284348.495984] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1285300.045973] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1285329.166973] Lustre: fir-MDT0002: haven't heard from client 58a1fac0-9465-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba4c5287800, cur 1585839871 expire 1585839721 last 1585839644 [1285339.803728] Lustre: fir-MDT0002: Connection restored to 21c8c585-5f1e-4 (at 10.50.8.18@o2ib2) [1286245.213766] Lustre: fir-MDT0002: haven't heard from client 0b982012-b4a2-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9217bbe000, cur 1585840787 expire 1585840637 last 1585840560 [1286332.760053] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1298821.500560] Lustre: fir-MDT0002: haven't heard from client 74a22577-dbc5-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b942eb36c00, cur 1585853363 expire 1585853213 last 1585853136 [1298851.039030] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1299220.054403] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1299280.513220] Lustre: fir-MDT0002: haven't heard from client 34f0ba71-4d3e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba460e800, cur 1585853822 expire 1585853672 last 1585853595 [1299774.524606] Lustre: fir-MDT0002: haven't heard from client a6f39580-441f-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ab243f000, cur 1585854316 expire 1585854166 last 1585854089 [1299840.859361] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1300674.996737] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1300721.549086] Lustre: fir-MDT0002: haven't heard from client ab887596-2654-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9748ec3c00, cur 1585855263 expire 1585855113 last 1585855036 [1301609.026074] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1301630.571090] Lustre: fir-MDT0002: haven't heard from client 7800f108-edbf-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9b962c0000, cur 1585856172 expire 1585856022 last 1585855945 [1302137.588679] Lustre: fir-MDT0002: haven't heard from client 25db87c4-882a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b48bdf000, cur 1585856679 expire 1585856529 last 1585856452 [1302388.281211] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1302708.992970] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1302767.601095] Lustre: fir-MDT0002: haven't heard from client b4ff50cd-f32a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b82087a8800, cur 1585857309 expire 1585857159 last 1585857082 [1304116.638405] Lustre: fir-MDT0002: haven't heard from client ddb48759-c809-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8ab82cc000, cur 1585858658 expire 1585858508 last 1585858431 [1304143.386259] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1304860.853908] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1304898.653926] Lustre: fir-MDT0002: haven't heard from client 4e2e9ff3-63e1-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8dc229c000, cur 1585859440 expire 1585859290 last 1585859213 [1305802.108560] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1305816.678748] Lustre: fir-MDT0002: haven't heard from client 40c9ab89-a8eb-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8dc229dc00, cur 1585860358 expire 1585860208 last 1585860131 [1306657.716744] Lustre: fir-MDT0002: haven't heard from client 0d59efc3-4437-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baa1dbacc00, cur 1585861199 expire 1585861049 last 1585860972 [1306684.456692] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1307615.723610] Lustre: fir-MDT0002: haven't heard from client 74c0ee2c-8abf-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8643aaf000, cur 1585862157 expire 1585862007 last 1585861930 [1307640.092416] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1308445.748547] Lustre: fir-MDT0002: haven't heard from client 7f24d805-4b6c-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8835eb9000, cur 1585862987 expire 1585862837 last 1585862760 [1308488.450366] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1309517.362140] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1309519.777690] Lustre: fir-MDT0002: haven't heard from client dae56af1-c8a1-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab98299400, cur 1585864061 expire 1585863911 last 1585863834 [1310397.793557] Lustre: fir-MDT0002: haven't heard from client 5582702c-e427-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b82087a9800, cur 1585864939 expire 1585864789 last 1585864712 [1310417.298882] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1311098.592543] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1311147.818839] Lustre: fir-MDT0002: haven't heard from client a5eeac10-067e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9a7fa0bc00, cur 1585865689 expire 1585865539 last 1585865462 [1311821.260627] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1311853.836226] Lustre: fir-MDT0002: haven't heard from client e22e23c2-e0bf-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b912fba4000, cur 1585866395 expire 1585866245 last 1585866168 [1312800.858106] Lustre: fir-MDT0002: haven't heard from client 8dc15a3d-db81-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b2f767000, cur 1585867342 expire 1585867192 last 1585867115 [1312862.599292] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1313716.883953] Lustre: fir-MDT0002: haven't heard from client 5124239a-c213-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9217bbbc00, cur 1585868258 expire 1585868108 last 1585868031 [1313737.416893] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1314501.640483] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1314516.904752] Lustre: fir-MDT0002: haven't heard from client c867a4b7-7672-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7dd0bd7c00, cur 1585869058 expire 1585868908 last 1585868831 [1315266.837947] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1315305.922525] Lustre: fir-MDT0002: haven't heard from client 08103f19-de06-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b82087af000, cur 1585869847 expire 1585869697 last 1585869620 [1316151.129904] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1316197.944710] Lustre: fir-MDT0002: haven't heard from client 69fe9fc9-3326-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba40261ac00, cur 1585870739 expire 1585870589 last 1585870512 [1317017.130147] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1317045.632575] LustreError: 41477:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.49.21.21@o2ib1) returned error from blocking AST (req@ffff8b855c702880 x1661551651296512 status -107 rc -107), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b8834e4d100/0x2f21cf25a70360d8 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 179 type: IBT flags: 0x60200400000020 nid: 10.49.21.21@o2ib1 remote: 0xc67cf33f077086a8 expref: 5 pid: 41531 timeout: 1317162 lvb_type: 0 [1317045.675870] LustreError: 138-a: fir-MDT0002: A client on nid 10.49.21.21@o2ib1 was evicted due to a lock blocking callback time out: rc -107 [1317045.688669] LustreError: 20500:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.49.21.21@o2ib1 ns: mdt-fir-MDT0002_UUID lock: ffff8b8834e4d100/0x2f21cf25a70360d8 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x5:0x0].0x0 bits 0x13/0x0 rrc: 177 type: IBT flags: 0x60200400000020 nid: 10.49.21.21@o2ib1 remote: 0xc67cf33f077086a8 expref: 6 pid: 41531 timeout: 0 lvb_type: 0 [1317237.970296] Lustre: fir-MDT0002: haven't heard from client 9fafd4c0-0b7b-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b728c942800, cur 1585871779 expire 1585871629 last 1585871552 [1317289.518340] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1317779.349867] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1317797.987311] Lustre: fir-MDT0002: haven't heard from client c65fab57-35bd-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b5645b000, cur 1585872339 expire 1585872189 last 1585872112 [1318566.080997] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1318585.016193] Lustre: fir-MDT0002: haven't heard from client 2aa88a3c-b748-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b924df25400, cur 1585873126 expire 1585872976 last 1585872899 [1319165.353766] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1319171.025688] Lustre: fir-MDT0002: haven't heard from client 5fea1caa-a7bb-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb3057c00, cur 1585873712 expire 1585873562 last 1585873485 [1319760.421015] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1319795.044342] Lustre: fir-MDT0002: haven't heard from client 4fc7f937-8bf2-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baaedb00800, cur 1585874336 expire 1585874186 last 1585874109 [1320666.061609] Lustre: fir-MDT0002: haven't heard from client 3abc36c6-05e4-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ba5866400, cur 1585875207 expire 1585875057 last 1585874980 [1320768.211128] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1321351.521606] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1321373.077177] Lustre: fir-MDT0002: haven't heard from client 457a238b-523a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b52a1e000, cur 1585875914 expire 1585875764 last 1585875687 [1321981.094249] Lustre: fir-MDT0002: haven't heard from client f8ebc65b-1d76-4 (at 10.49.21.13@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb9229c00, cur 1585876522 expire 1585876372 last 1585876295 [1322057.469402] Lustre: fir-MDT0002: Connection restored to f8ebc65b-1d76-4 (at 10.49.21.13@o2ib1) [1322381.105259] Lustre: fir-MDT0002: haven't heard from client cb3fb692-e931-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9217bbf000, cur 1585876922 expire 1585876772 last 1585876695 [1322463.034171] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1328537.264615] Lustre: fir-MDT0002: haven't heard from client 0317481a-dbc1-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b488dc000, cur 1585883078 expire 1585882928 last 1585882851 [1328603.147583] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1347108.184068] Lustre: fir-MDT0002: Connection restored to 06f6e79a-f25a-4 (at 10.49.23.21@o2ib1) [1347168.733323] Lustre: fir-MDT0002: haven't heard from client 06f6e79a-f25a-4 (at 10.49.23.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb2666800, cur 1585901709 expire 1585901559 last 1585901482 [1366058.382769] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1366088.218858] Lustre: fir-MDT0002: haven't heard from client e11b8b3f-72d0-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9b83160400, cur 1585920628 expire 1585920478 last 1585920401 [1366792.900790] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1366838.239065] Lustre: fir-MDT0002: haven't heard from client d474ebd5-a9fc-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba2696f3c00, cur 1585921378 expire 1585921228 last 1585921151 [1367749.281904] Lustre: fir-MDT0002: haven't heard from client e2f38f60-659f-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8eb9500800, cur 1585922289 expire 1585922139 last 1585922062 [1367760.972819] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1368465.706196] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1368516.282304] Lustre: fir-MDT0002: haven't heard from client 46b016f2-d1f4-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b4de23c00, cur 1585923056 expire 1585922906 last 1585922829 [1368592.287476] Lustre: fir-MDT0002: haven't heard from client 681deff3-eb13-4 (at 10.50.6.54@o2ib2) in 202 seconds. I think it's dead, and I am evicting it. exp ffff8b8b9c974c00, cur 1585923132 expire 1585922982 last 1585922930 [1368665.123518] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1369396.307699] Lustre: fir-MDT0002: haven't heard from client 7d2ba658-0baa-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9b85d13400, cur 1585923936 expire 1585923786 last 1585923709 [1369459.080314] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1370239.367432] Lustre: fir-MDT0002: haven't heard from client f0184f56-3555-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9b85d12000, cur 1585924779 expire 1585924629 last 1585924552 [1370271.851170] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1371146.986464] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1371151.358700] Lustre: fir-MDT0002: haven't heard from client 658cd963-8a05-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9748f96000, cur 1585925691 expire 1585925541 last 1585925464 [1371889.707758] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1371952.369357] Lustre: fir-MDT0002: haven't heard from client e0d2a6f9-90c8-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8ba6efa800, cur 1585926492 expire 1585926342 last 1585926265 [1372741.761094] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1372770.402513] Lustre: fir-MDT0002: haven't heard from client 67ebe6f5-5aae-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ba5860400, cur 1585927310 expire 1585927160 last 1585927083 [1373747.417557] Lustre: fir-MDT0002: haven't heard from client 996bbcc9-9596-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b98c3f1f800, cur 1585928287 expire 1585928137 last 1585928060 [1373769.958710] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1374548.789292] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1374575.439769] Lustre: fir-MDT0002: haven't heard from client 62b6f27c-0c27-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8bb5955800, cur 1585929115 expire 1585928965 last 1585928888 [1375455.160657] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1375479.487926] Lustre: fir-MDT0002: haven't heard from client 315777c7-4f4b-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8bb000b000, cur 1585930019 expire 1585929869 last 1585929792 [1376279.096679] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1376310.479024] Lustre: fir-MDT0002: haven't heard from client 1adc3b55-8749-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9bac42f000, cur 1585930850 expire 1585930700 last 1585930623 [1376419.065880] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1377128.169990] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1377159.556696] Lustre: fir-MDT0002: haven't heard from client a9261f64-2c5e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7e946bac00, cur 1585931699 expire 1585931549 last 1585931472 [1377159.576844] Lustre: Skipped 1 previous similar message [1378008.520873] Lustre: fir-MDT0002: haven't heard from client a93e6deb-131a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9bac42c800, cur 1585932548 expire 1585932398 last 1585932321 [1378033.269718] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1378065.638328] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1378764.054766] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1378813.544233] Lustre: fir-MDT0002: haven't heard from client f414c76d-7b01-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab4d80e400, cur 1585933353 expire 1585933203 last 1585933126 [1378813.564396] Lustre: Skipped 1 previous similar message [1379723.565000] Lustre: fir-MDT0002: haven't heard from client 0ec25327-8576-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7bae9f9000, cur 1585934263 expire 1585934113 last 1585934036 [1379770.567583] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1379875.680889] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1379920.571873] Lustre: fir-MDT0002: haven't heard from client a0f2210c-95b3-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab4cfe9c00, cur 1585934460 expire 1585934310 last 1585934233 [1381049.066374] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1381107.600469] Lustre: fir-MDT0002: haven't heard from client 8eae559f-8716-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba7def49400, cur 1585935647 expire 1585935497 last 1585935420 [1381963.258293] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1381979.625121] Lustre: fir-MDT0002: haven't heard from client 48846965-459c-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8eb9504c00, cur 1585936519 expire 1585936369 last 1585936292 [1382432.632138] Lustre: fir-MDT0002: haven't heard from client c1177a89-5194-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7ba13b4000, cur 1585936972 expire 1585936822 last 1585936745 [1382469.582606] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1382591.689906] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1382643.636423] Lustre: fir-MDT0002: haven't heard from client 304e6b7a-f74b-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b49a74400, cur 1585937183 expire 1585937033 last 1585936956 [1383548.011822] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1383597.660385] Lustre: fir-MDT0002: haven't heard from client e26a0b8d-df3e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b9c97ec00, cur 1585938137 expire 1585937987 last 1585937910 [1384354.683988] Lustre: fir-MDT0002: haven't heard from client 539d87e3-6b19-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b5b26e000, cur 1585938894 expire 1585938744 last 1585938667 [1384396.624201] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1386430.729232] Lustre: fir-MDT0002: haven't heard from client 835afd83-dfec-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b55b6b000, cur 1585940970 expire 1585940820 last 1585940743 [1386475.658164] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1389438.802856] Lustre: fir-MDT0002: haven't heard from client 5ab99ad1-8b26-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7872f44000, cur 1585943978 expire 1585943828 last 1585943751 [1389664.614186] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1389697.808019] Lustre: fir-MDT0002: haven't heard from client 7f196257-4021-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b48905800, cur 1585944237 expire 1585944087 last 1585944010 [1389702.875900] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1389874.209970] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1389929.838182] Lustre: fir-MDT0002: haven't heard from client 3f36eb03-79bd-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b49002400, cur 1585944469 expire 1585944319 last 1585944242 [1390049.199316] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1390127.819535] Lustre: fir-MDT0002: haven't heard from client 274a7547-b7aa-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b92c3293800, cur 1585944667 expire 1585944517 last 1585944440 [1391799.869545] Lustre: fir-MDT0002: haven't heard from client 73f503d3-884e-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b92c3290400, cur 1585946339 expire 1585946189 last 1585946112 [1391849.063176] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1393557.907210] Lustre: fir-MDT0002: haven't heard from client 75373d48-fe8f-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8bb942a400, cur 1585948097 expire 1585947947 last 1585947870 [1393598.586014] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1397188.998899] Lustre: fir-MDT0002: haven't heard from client 69cc2ce3-9c83-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b55dbb000, cur 1585951728 expire 1585951578 last 1585951501 [1397246.704923] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1423642.672230] Lustre: fir-MDT0002: haven't heard from client 037ca474-0477-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6baaf92c00, cur 1585978181 expire 1585978031 last 1585977954 [1423691.107050] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1436282.989430] Lustre: fir-MDT0002: haven't heard from client cc32a056-ed4a-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8dacbc3400, cur 1585990821 expire 1585990671 last 1585990594 [1436312.865786] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1437770.027390] Lustre: fir-MDT0002: haven't heard from client e4687612-d8e5-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7f35342400, cur 1585992308 expire 1585992158 last 1585992081 [1437807.989999] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1441954.580319] Lustre: fir-MDT0002: Client c742d3fe-d253-4 (at 10.49.8.33@o2ib1) reconnecting [1441954.589037] Lustre: Skipped 4 previous similar messages [1441954.594727] Lustre: fir-MDT0002: Connection restored to c742d3fe-d253-4 (at 10.49.8.33@o2ib1) [1442924.544416] Lustre: fir-MDT0002: Client d3db911c-747e-4 (at 10.49.0.62@o2ib1) reconnecting [1442924.552864] Lustre: Skipped 2 previous similar messages [1442924.558295] Lustre: fir-MDT0002: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [1442924.566999] Lustre: Skipped 2 previous similar messages [1445186.108000] Lustre: fir-MDT0002: Connection restored to bb05bfe6-c379-4 (at 10.50.4.28@o2ib2) [1445208.212501] Lustre: fir-MDT0002: haven't heard from client bb05bfe6-c379-4 (at 10.50.4.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb5085000, cur 1585999746 expire 1585999596 last 1585999519 [1449782.667225] Lustre: fir-MDT0002: Connection restored to bb05bfe6-c379-4 (at 10.50.4.28@o2ib2) [1449827.330662] Lustre: fir-MDT0002: haven't heard from client 9b7d199c-0123-4 (at 10.50.4.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba7883f9400, cur 1586004365 expire 1586004215 last 1586004138 [1451660.374487] Lustre: fir-MDT0002: haven't heard from client 934ddb45-94a2-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b9d5bf800, cur 1586006198 expire 1586006048 last 1586005971 [1451696.450237] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1452405.058316] Lustre: fir-MDT0002: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1452466.394346] Lustre: fir-MDT0002: haven't heard from client 68109525-701f-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab4af93400, cur 1586007004 expire 1586006854 last 1586006777 [1456358.493774] Lustre: fir-MDT0002: haven't heard from client 3dd153d6-2947-4 (at 10.50.4.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7bae9fd000, cur 1586010896 expire 1586010746 last 1586010669 [1456747.601979] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1456775.502031] Lustre: fir-MDT0002: haven't heard from client 0cb0db66-c8ed-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baa0921c800, cur 1586011313 expire 1586011163 last 1586011086 [1457327.520040] Lustre: fir-MDT0002: haven't heard from client 9b0c99d2-345c-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9b82eb1c00, cur 1586011865 expire 1586011715 last 1586011638 [1457375.783365] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1457713.297108] Lustre: fir-MDT0002: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1457751.527547] Lustre: fir-MDT0002: haven't heard from client ca6aa2c0-a51a-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b9f2f1800, cur 1586012289 expire 1586012139 last 1586012062 [1457799.807002] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1457827.530186] Lustre: fir-MDT0002: haven't heard from client 33c56caf-f05e-4 (at 10.49.21.21@o2ib1) in 199 seconds. I think it's dead, and I am evicting it. exp ffff8b9ab528ec00, cur 1586012365 expire 1586012215 last 1586012166 [1460038.130650] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1460060.586112] Lustre: fir-MDT0002: haven't heard from client f072dd11-76e6-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b98f12d6c00, cur 1586014598 expire 1586014448 last 1586014371 [1460136.588405] Lustre: fir-MDT0002: haven't heard from client 5a471ce2-27f1-4 (at 10.50.6.54@o2ib2) in 176 seconds. I think it's dead, and I am evicting it. exp ffff8b89d7b18400, cur 1586014674 expire 1586014524 last 1586014498 [1460283.143325] Lustre: fir-MDT0002: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1462674.649534] Lustre: fir-MDT0002: haven't heard from client d6539a1b-0c76-4 (at 10.49.7.8@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaa2e6400, cur 1586017212 expire 1586017062 last 1586016985 [1462740.911766] Lustre: fir-MDT0002: Connection restored to d6539a1b-0c76-4 (at 10.49.7.8@o2ib1) [1463551.673229] Lustre: fir-MDT0002: haven't heard from client 97017afb-9fdb-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7272450c00, cur 1586018089 expire 1586017939 last 1586017862 [1463578.806517] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1474452.811384] Lustre: fir-MDT0002: Connection restored to ee56fbe2-040d-4 (at 10.49.25.17@o2ib1) [1478868.049625] Lustre: fir-MDT0002: haven't heard from client ff3d8c7f-ee13-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b78741b0c00, cur 1586033405 expire 1586033255 last 1586033178 [1478902.370193] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1483478.487361] Lustre: fir-MDT0002: Client 77e2c13e-26fe-4 (at 10.49.7.20@o2ib1) reconnecting [1483478.495831] Lustre: fir-MDT0002: Connection restored to 77e2c13e-26fe-4 (at 10.49.7.20@o2ib1) [1484450.000291] Lustre: fir-MDT0002: Client c742d3fe-d253-4 (at 10.49.8.33@o2ib1) reconnecting [1484450.008729] Lustre: Skipped 1 previous similar message [1484450.014070] Lustre: fir-MDT0002: Connection restored to c742d3fe-d253-4 (at 10.49.8.33@o2ib1) [1484450.022769] Lustre: Skipped 1 previous similar message [1485154.005639] Lustre: fir-MDT0002: Client 49c076ac-79a3-4 (at 10.49.18.29@o2ib1) reconnecting [1485154.014184] Lustre: fir-MDT0002: Connection restored to 49c076ac-79a3-4 (at 10.49.18.29@o2ib1) [1486357.640249] Lustre: fir-MDT0002: Client 6470996b-3104-4 (at 10.49.7.14@o2ib1) reconnecting [1486357.648718] Lustre: fir-MDT0002: Connection restored to 6470996b-3104-4 (at 10.49.7.14@o2ib1) [1486452.460419] Lustre: fir-MDT0002: Client 9c05561a-1cd1-4 (at 10.49.8.19@o2ib1) reconnecting [1486452.468887] Lustre: fir-MDT0002: Connection restored to 63f2b4c1-0051-4 (at 10.49.8.19@o2ib1) [1486693.820808] Lustre: fir-MDT0002: Client 0a9b0342-b987-4 (at 10.49.7.19@o2ib1) reconnecting [1486693.829279] Lustre: fir-MDT0002: Connection restored to 0a9b0342-b987-4 (at 10.49.7.19@o2ib1) [1486962.297094] Lustre: fir-MDT0002: Client 4cb83747-634b-4 (at 10.49.8.26@o2ib1) reconnecting [1486962.297168] Lustre: fir-MDT0002: Connection restored to e0f8d55d-8a95-4 (at 10.49.8.21@o2ib1) [1486962.297170] Lustre: Skipped 1 previous similar message [1486962.319564] Lustre: Skipped 2 previous similar messages [1489396.310521] Lustre: fir-MDT0002: haven't heard from client dfbfebbc-fbbb-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab4ce4b000, cur 1586043933 expire 1586043783 last 1586043706 [1489432.261813] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1489432.270519] Lustre: Skipped 1 previous similar message [1503952.667359] Lustre: fir-MDT0002: haven't heard from client 03fdb3f2-9fe5-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6ba4fd3000, cur 1586058489 expire 1586058339 last 1586058262 [1504019.607338] Lustre: fir-MDT0002: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [1504258.674152] Lustre: fir-MDT0002: haven't heard from client af8b075a-d960-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7bb9201800, cur 1586058795 expire 1586058645 last 1586058568 [1504283.664718] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1504468.678589] Lustre: fir-MDT0002: haven't heard from client 05c68320-d699-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b3625a000, cur 1586059005 expire 1586058855 last 1586058778 [1504505.354982] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1506566.731416] Lustre: fir-MDT0002: haven't heard from client ce295894-cf8f-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b55b69800, cur 1586061103 expire 1586060953 last 1586060876 [1506676.339055] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1508833.784726] Lustre: fir-MDT0002: haven't heard from client 87e3819d-30dd-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9bba3b8800, cur 1586063370 expire 1586063220 last 1586063143 [1508869.989030] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1511205.842931] Lustre: fir-MDT0002: haven't heard from client d497665e-1a35-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7f35341000, cur 1586065742 expire 1586065592 last 1586065515 [1511253.577981] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1518035.011448] Lustre: fir-MDT0002: haven't heard from client 8d22f512-aa17-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b92c3290000, cur 1586072571 expire 1586072421 last 1586072344 [1518076.825392] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1518431.020930] Lustre: fir-MDT0002: haven't heard from client e51c3087-0e32-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b83e5bb9800, cur 1586072967 expire 1586072817 last 1586072740 [1518499.494120] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1520960.989008] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586075489/real 1586075489] req@ffff8bab83f04380 x1661552638999808/t0(0) o104->fir-MDT0002@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586075496 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1520968.016185] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586075496/real 1586075496] req@ffff8bab83f04380 x1661552638999808/t0(0) o104->fir-MDT0002@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586075503 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1520975.043358] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586075503/real 1586075503] req@ffff8bab83f04380 x1661552638999808/t0(0) o104->fir-MDT0002@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586075510 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1520982.070528] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586075510/real 1586075510] req@ffff8bab83f04380 x1661552638999808/t0(0) o104->fir-MDT0002@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586075517 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1520996.097871] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586075525/real 1586075525] req@ffff8bab83f04380 x1661552638999808/t0(0) o104->fir-MDT0002@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586075532 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1520996.125303] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [1521017.137388] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586075546/real 1586075546] req@ffff8bab83f04380 x1661552638999808/t0(0) o104->fir-MDT0002@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586075553 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1521017.164814] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [1521052.176297] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586075581/real 1586075581] req@ffff8bab83f04380 x1661552638999808/t0(0) o104->fir-MDT0002@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586075588 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1521052.203730] Lustre: 41360:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [1521108.219714] LustreError: 41360:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.9.27@o2ib2) failed to reply to blocking AST (req@ffff8bab83f04380 x1661552638999808 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b8f10540fc0/0x2f21cf2802f36747 lrc: 4/0,0 mode: PR/PR res: [0x2c0039230:0x84ca:0x0].0x0 bits 0x13/0x0 rrc: 372 type: IBT flags: 0x60200400000020 nid: 10.50.9.27@o2ib2 remote: 0x74eb7d302500919e expref: 14 pid: 41351 timeout: 1521212 lvb_type: 0 [1521108.262851] LustreError: 138-a: fir-MDT0002: A client on nid 10.50.9.27@o2ib2 was evicted due to a lock blocking callback time out: rc -110 [1521108.275561] LustreError: 20500:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 155s: evicting client at 10.50.9.27@o2ib2 ns: mdt-fir-MDT0002_UUID lock: ffff8b8f10540fc0/0x2f21cf2802f36747 lrc: 3/0,0 mode: PR/PR res: [0x2c0039230:0x84ca:0x0].0x0 bits 0x13/0x0 rrc: 372 type: IBT flags: 0x60200400000020 nid: 10.50.9.27@o2ib2 remote: 0x74eb7d302500919e expref: 15 pid: 41351 timeout: 0 lvb_type: 0 [1521195.205730] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1523455.145588] Lustre: fir-MDT0002: haven't heard from client a734e045-6ae0-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b4de21800, cur 1586077991 expire 1586077841 last 1586077764 [1523477.166950] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1529302.289631] Lustre: fir-MDT0002: haven't heard from client 3ac90f8e-5504-4 (at 10.49.7.8@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ab17fc400, cur 1586083838 expire 1586083688 last 1586083611 [1533034.342516] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586087562/real 1586087562] req@ffff8b9607e59f80 x1661552780602560/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586087569 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1533034.370032] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [1533048.379852] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586087577/real 1586087577] req@ffff8b9607e59f80 x1661552780602560/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586087584 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1533048.407382] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [1533069.417368] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586087598/real 1586087598] req@ffff8b9607e59f80 x1661552780602560/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586087605 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1533069.444901] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [1533104.455227] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586087633/real 1586087633] req@ffff8b9607e59f80 x1661552780602560/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586087640 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1533104.482755] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [1533174.494943] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586087703/real 1586087703] req@ffff8b9607e59f80 x1661552780602560/t0(0) o104->fir-MDT0002@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586087710 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1533174.522461] Lustre: 124966:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [1533181.532135] LustreError: 124966:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.9.37@o2ib2) failed to reply to blocking AST (req@ffff8b9607e59f80 x1661552780602560 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8baa90b3b180/0x2f21cf283287b280 lrc: 4/0,0 mode: PR/PR res: [0x2c0039230:0x84ca:0x0].0x0 bits 0x13/0x0 rrc: 367 type: IBT flags: 0x60200400000020 nid: 10.50.9.37@o2ib2 remote: 0x3787d70896c69e01 expref: 161 pid: 20921 timeout: 1533285 lvb_type: 0 [1533181.575455] LustreError: 138-a: fir-MDT0002: A client on nid 10.50.9.37@o2ib2 was evicted due to a lock blocking callback time out: rc -110 [1533181.588170] LustreError: 20500:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 155s: evicting client at 10.50.9.37@o2ib2 ns: mdt-fir-MDT0002_UUID lock: ffff8baa90b3b180/0x2f21cf283287b280 lrc: 3/0,0 mode: PR/PR res: [0x2c0039230:0x84ca:0x0].0x0 bits 0x13/0x0 rrc: 367 type: IBT flags: 0x60200400000020 nid: 10.50.9.37@o2ib2 remote: 0x3787d70896c69e01 expref: 162 pid: 20921 timeout: 0 lvb_type: 0 [1533262.234250] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1536862.475030] Lustre: fir-MDT0002: haven't heard from client 5c81c5e2-1ff2-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba4363d6000, cur 1586091398 expire 1586091248 last 1586091171 [1537227.738049] Lustre: fir-MDT0002: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [1543977.648902] Lustre: fir-MDT0002: haven't heard from client 29f566cd-115e-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8643aa8400, cur 1586098513 expire 1586098363 last 1586098286 [1544021.726034] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1562437.104551] Lustre: fir-MDT0002: haven't heard from client 90b6f92d-d059-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ba96de24000, cur 1586116972 expire 1586116822 last 1586116745 [1562656.025162] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1565689.221410] Lustre: fir-MDT0002: Client 0ac890ab-ac29-4 (at 10.50.4.25@o2ib2) reconnecting [1565689.229873] Lustre: fir-MDT0002: Connection restored to 0ac890ab-ac29-4 (at 10.50.4.25@o2ib2) [1568750.366729] Lustre: fir-MDT0002: Client 14f6955a-e3d1-4 (at 10.49.8.32@o2ib1) reconnecting [1568750.375194] Lustre: fir-MDT0002: Connection restored to 14f6955a-e3d1-4 (at 10.49.8.32@o2ib1) [1568751.766972] Lustre: fir-MDT0002: Client 20917503-c5d6-4 (at 10.49.18.34@o2ib1) reconnecting [1568751.775526] Lustre: fir-MDT0002: Connection restored to 20917503-c5d6-4 (at 10.49.18.34@o2ib1) [1568751.784316] Lustre: Skipped 1 previous similar message [1568752.773433] Lustre: fir-MDT0002: Client cc51ef17-5ec1-4 (at 10.49.17.18@o2ib1) reconnecting [1568752.781963] Lustre: Skipped 9 previous similar messages [1568752.787387] Lustre: fir-MDT0002: Connection restored to cc51ef17-5ec1-4 (at 10.49.17.18@o2ib1) [1568752.796175] Lustre: Skipped 8 previous similar messages [1568754.788219] Lustre: fir-MDT0002: Client 051aface-8044-4 (at 10.49.27.28@o2ib1) reconnecting [1568754.795521] Lustre: fir-MDT0002: Connection restored to 01256051-b408-4 (at 10.49.23.30@o2ib1) [1568754.795523] Lustre: Skipped 55 previous similar messages [1568754.811020] Lustre: Skipped 58 previous similar messages [1568758.826764] Lustre: fir-MDT0002: Client ab903916-fdad-4 (at 10.49.30.5@o2ib1) reconnecting [1568758.835213] Lustre: Skipped 198 previous similar messages [1568758.840810] Lustre: fir-MDT0002: Connection restored to ab903916-fdad-4 (at 10.49.30.5@o2ib1) [1568758.849536] Lustre: Skipped 201 previous similar messages [1568767.458710] Lustre: fir-MDT0002: Client 3a21098c-107c-4 (at 10.49.7.18@o2ib1) reconnecting [1568767.467153] Lustre: Skipped 54 previous similar messages [1568767.472666] Lustre: fir-MDT0002: Connection restored to 3a21098c-107c-4 (at 10.49.7.18@o2ib1) [1568767.481364] Lustre: Skipped 53 previous similar messages [1568852.057584] Lustre: fir-MDT0002: Client 42c9b425-8bb9-4 (at 10.49.30.14@o2ib1) reconnecting [1568852.066115] Lustre: Skipped 3 previous similar messages [1568852.071551] Lustre: fir-MDT0002: Connection restored to 42c9b425-8bb9-4 (at 10.49.30.14@o2ib1) [1568852.080338] Lustre: Skipped 3 previous similar messages [1586820.711718] Lustre: fir-MDT0002: haven't heard from client 89b8930d-7d49-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b49007800, cur 1586141355 expire 1586141205 last 1586141128 [1586857.924382] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1586857.933087] Lustre: Skipped 317 previous similar messages [1590582.801241] Lustre: fir-MDT0002: haven't heard from client fe9d3211-9fa0-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b98c3f1a800, cur 1586145117 expire 1586144967 last 1586144890 [1590625.207223] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1614514.615320] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1614543.414983] Lustre: fir-MDT0002: haven't heard from client 5b0cb4e2-0c62-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab51261000, cur 1586169077 expire 1586168927 last 1586168850 [1623950.641908] Lustre: fir-MDT0002: haven't heard from client b9dc3cb9-ba1a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9aca08cc00, cur 1586178484 expire 1586178334 last 1586178257 [1624037.747016] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1625433.747400] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1625470.679358] Lustre: fir-MDT0002: haven't heard from client 5d276014-2b20-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9adb077800, cur 1586180004 expire 1586179854 last 1586179777 [1639449.025607] Lustre: fir-MDT0002: haven't heard from client 3367b05d-b386-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab4af8f400, cur 1586193982 expire 1586193832 last 1586193755 [1639506.303245] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1639736.297758] Lustre: fir-MDT0002: Client 980e752f-05e5-4 (at 10.50.7.37@o2ib2) reconnecting [1639736.306203] Lustre: Skipped 317 previous similar messages [1639736.311805] Lustre: fir-MDT0002: Connection restored to 980e752f-05e5-4 (at 10.50.7.37@o2ib2) [1647305.789398] Lustre: fir-MDT0002: Connection restored to d6539a1b-0c76-4 (at 10.49.7.8@o2ib1) [1653462.898514] Lustre: fir-MDT0002: Connection restored to bb05bfe6-c379-4 (at 10.50.4.28@o2ib2) [1654735.961631] Lustre: fir-MDT0002: Connection restored to 9823016a-d5f7-4 (at 10.49.0.63@o2ib1) [1654788.406666] Lustre: fir-MDT0002: haven't heard from client 9823016a-d5f7-4 (at 10.49.0.63@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba4ea0800, cur 1586209321 expire 1586209171 last 1586209094 [1714554.405785] Lustre: fir-MDT0002: Connection restored to b3a54002-f8c5-4 (at 10.50.12.2@o2ib2) [1714576.910944] Lustre: fir-MDT0002: haven't heard from client b3a54002-f8c5-4 (at 10.50.12.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb5086c00, cur 1586269108 expire 1586268958 last 1586268881 [1722160.631546] Lustre: fir-MDT0002: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1722178.099238] Lustre: fir-MDT0002: haven't heard from client 19ab4ca3-4eb4-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ba5866000, cur 1586276709 expire 1586276559 last 1586276482 [1725687.609945] Lustre: fir-MDT0002: Connection restored to b3a54002-f8c5-4 (at 10.50.12.2@o2ib2) [1727808.244660] Lustre: fir-MDT0002: haven't heard from client 7c15878b-7f74-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b4a635400, cur 1586282339 expire 1586282189 last 1586282112 [1727841.428878] Lustre: fir-MDT0002: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1746081.691886] Lustre: fir-MDT0002: haven't heard from client eeb6444d-4e51-4 (at 10.50.16.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ab166e000, cur 1586300612 expire 1586300462 last 1586300385 [1746102.994713] Lustre: fir-MDT0002: Connection restored to 6c45c03c-4b15-4 (at 10.50.16.6@o2ib2) [1757961.998414] Lustre: fir-MDT0002: haven't heard from client 2ad6fa60-c8a2-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9acf169800, cur 1586312492 expire 1586312342 last 1586312265 [1758011.058027] Lustre: fir-MDT0002: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1760923.558497] Lustre: 20330:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586314851/real 1586314851] req@ffff8b8824a0d100 x1661556059926656/t0(0) o6->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 8 to 1 dl 1586315452 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1760923.586786] Lustre: 20330:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [1760923.596623] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1760923.613146] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1761187.831349] Lustre: 20702:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315116/real 1586315116] req@ffff8b7bb04a3600 x1661556068431232/t0(0) o5->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 432/432 e 2 to 1 dl 1586315717 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [1761187.859744] LustreError: 20702:0:(osp_precreate.c:686:osp_precreate_send()) fir-OST0034-osc-MDT0002: can't precreate: rc = -107 [1761281.182778] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315453/real 1586315453] req@ffff8b854e6f8480 x1661556068052736/t0(0) o6->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 1 to 1 dl 1586315811 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1761281.211272] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1761281.227750] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1761640.048117] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315811/real 1586315811] req@ffff8b854e6f8480 x1661556068052736/t0(0) o6->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 1 to 1 dl 1586316169 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1761640.076586] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1761640.093060] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1761943.892049] Lustre: 20702:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315717/real 1586315717] req@ffff8b7bb04a0900 x1661556082209536/t0(0) o5->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 432/432 e 0 to 1 dl 1586316473 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [1761943.920436] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -107 [1761998.601532] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1761998.618038] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1762049.665859] Lustre: fir-OST003a-osc-MDT0002: Connection to fir-OST003a (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1762049.682392] Lustre: fir-OST003a-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1762356.210913] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586316528/real 1586316528] req@ffff8b854e6f8480 x1661556068052736/t0(0) o6->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 1 to 1 dl 1586316886 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1762356.239376] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [1762356.249296] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1762356.265798] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1762382.643603] Lustre: fir-OST0032-osc-MDT0002: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1762382.660057] Lustre: fir-OST0032-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1762597.744090] Lustre: fir-MDT0002: Connection restored to 9ed38912-482b-4 (at 10.50.1.57@o2ib2) [1762660.141767] Lustre: fir-MDT0002: haven't heard from client 9ed38912-482b-4 (at 10.50.1.57@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb1710800, cur 1586317190 expire 1586317040 last 1586316963 [1762700.952798] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -107 [1762714.765148] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1762714.781650] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1762790.526070] Lustre: fir-OST0038-osc-MDT0002: Connection to fir-OST0038 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1762956.263009] Lustre: fir-MDT0002: Connection restored to 78da859a-079c-4 (at 10.50.4.27@o2ib2) [1762956.271710] Lustre: Skipped 2 previous similar messages [1763014.129472] Lustre: fir-MDT0002: haven't heard from client 2147e45e-5aee-4 (at 10.50.4.29@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba0241000, cur 1586317544 expire 1586317394 last 1586317317 [1763072.645281] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586317244/real 1586317244] req@ffff8b854e6f8480 x1661556068052736/t0(0) o6->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 1 to 1 dl 1586317602 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1763072.673744] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [1763072.683666] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1763072.699923] Lustre: Skipped 1 previous similar message [1763249.913889] Lustre: fir-OST0030-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1763249.923893] Lustre: Skipped 4 previous similar messages [1763430.358009] Lustre: fir-OST0034-osc-MDT0002: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1763430.374250] Lustre: Skipped 2 previous similar messages [1763457.984680] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -107 [1763678.145586] Lustre: fir-MDT0002: haven't heard from client 81fc576f-a47a-4 (at 10.50.10.31@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaa2e5400, cur 1586318208 expire 1586318058 last 1586317981 [1763678.165753] Lustre: Skipped 2 previous similar messages [1763788.806759] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586317960/real 1586317960] req@ffff8b854e6f8480 x1661556068052736/t0(0) o6->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 1 to 1 dl 1586318318 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1763788.835225] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [1763788.845382] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1763788.855385] Lustre: Skipped 4 previous similar messages [1764005.924064] Lustre: fir-OST0030-osc-MDT0002: Connection to fir-OST0030 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1764005.940304] Lustre: Skipped 4 previous similar messages [1764106.163019] Lustre: fir-MDT0002: haven't heard from client 376f46a0-8125-4 (at 10.50.5.18@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb212f000, cur 1586318636 expire 1586318486 last 1586318409 [1764215.017111] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -107 [1764505.496099] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586318676/real 1586318676] req@ffff8b854e6f8480 x1661556068052736/t0(0) o6->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 1 to 1 dl 1586319034 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1764505.524568] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [1764505.534741] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1764505.544762] Lustre: Skipped 6 previous similar messages [1764651.515607] Lustre: fir-OST0032-osc-MDT0002: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1764651.531852] Lustre: Skipped 4 previous similar messages [1764765.170994] Lustre: fir-MDT0002: haven't heard from client d2c2a701-05d0-4 (at 10.50.2.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb38ea800, cur 1586319295 expire 1586319145 last 1586319068 [1764972.048284] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -107 [1765074.208643] Lustre: fir-MDT0002: haven't heard from client c385efcd-dc92-4 (at 10.50.2.30@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bab9d745000, cur 1586319604 expire 1586319454 last 1586319377 [1765223.026424] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586319394/real 1586319394] req@ffff8b854e6f8480 x1661556068052736/t0(0) o6->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 1 to 1 dl 1586319752 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1765223.054889] Lustre: 20322:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [1765223.065112] Lustre: fir-OST0034-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1765223.075133] Lustre: Skipped 8 previous similar messages [1765407.757972] Lustre: fir-OST0032-osc-MDT0002: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1765407.774210] Lustre: Skipped 5 previous similar messages [1765729.079941] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -107 [1765829.816426] Lustre: 20329:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586319603/real 1586319603] req@ffff8b8a56ef8900 x1661556082361856/t0(0) o6->fir-OST003a-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 544/432 e 0 to 1 dl 1586320359 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1765829.844713] Lustre: 20329:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [1765829.854852] Lustre: fir-OST003a-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1765829.864881] Lustre: Skipped 4 previous similar messages [1766163.760704] Lustre: fir-OST0032-osc-MDT0002: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1766163.776943] Lustre: Skipped 5 previous similar messages [1766185.209449] Lustre: fir-MDT0002: haven't heard from client a95eaf77-b4b8-4 (at 10.50.2.59@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba8e48800, cur 1586320715 expire 1586320565 last 1586320488 [1766185.229512] Lustre: Skipped 1 previous similar message [1766486.111743] Lustre: 20702:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586320259/real 1586320259] req@ffff8b9bb2019680 x1661556188890048/t0(0) o5->fir-OST0034-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 432/432 e 0 to 1 dl 1586321015 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [1766486.140143] Lustre: 20702:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [1766486.150063] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -107 [1766571.227087] Lustre: fir-OST0038-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1766571.237095] Lustre: Skipped 5 previous similar messages [1766919.987556] Lustre: fir-OST0032-osc-MDT0002: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1766920.003799] Lustre: Skipped 5 previous similar messages [1767019.559038] LNetError: 20288:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [1767019.569217] LNetError: 20288:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.109@o2ib7 (30): c: 0, oc: 0, rc: 8 [1767019.581752] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [1767019.593860] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 10 previous similar messages [1767029.467542] Lustre: fir-MDT0002: Client 8c251876-2bc2-4 (at 10.50.10.12@o2ib2) reconnecting [1767030.530568] Lustre: fir-MDT0002: Client 16365f08-4eb8-4 (at 10.49.28.6@o2ib1) reconnecting [1767032.208392] Lustre: fir-MDT0002: Client a489af8f-6baa-4 (at 10.49.23.28@o2ib1) reconnecting [1767036.169138] Lustre: fir-MDT0002: Client 0f2d3178-05c2-4 (at 10.49.0.64@o2ib1) reconnecting [1767044.153925] Lustre: fir-MDT0002: Client 00c7f158-cc8b-4 (at 10.50.4.48@o2ib2) reconnecting [1767044.162365] Lustre: Skipped 2 previous similar messages [1767046.786250] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.13.13@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767054.145477] Lustre: fir-MDT0002: Client 0972a588-f607-4 (at 10.50.8.27@o2ib2) reconnecting [1767054.153923] Lustre: Skipped 5 previous similar messages [1767059.032847] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.10.38@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767063.243315] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.7.1@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767070.911561] Lustre: fir-MDT0002: Client 00d873ba-49ea-4 (at 10.50.2.61@o2ib2) reconnecting [1767070.920002] Lustre: Skipped 17 previous similar messages [1767072.560355] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 17 seconds [1767072.570790] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 12 previous similar messages [1767072.580367] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [1767072.592442] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message [1767074.179867] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.13.9@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767088.898108] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.14.3@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767088.915562] LustreError: Skipped 2 previous similar messages [1767089.558778] Lustre: 124960:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586321612/real 1586321612] req@ffff8b98d52a5100 x1661556190630144/t0(0) o104->fir-MDT0002@10.50.4.70@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1586321619 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1767089.586304] Lustre: 124960:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [1767099.229565] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.8.42@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767099.247023] LustreError: Skipped 1 previous similar message [1767103.238964] Lustre: fir-MDT0002: Client b2ca0981-ba7b-4 (at 10.50.15.6@o2ib2) reconnecting [1767103.247405] Lustre: Skipped 41 previous similar messages [1767118.408505] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.27.23@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767118.426046] LustreError: Skipped 4 previous similar messages [1767123.561630] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 9 seconds [1767123.571973] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 17 previous similar messages [1767123.581550] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [1767150.904911] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.6.12@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767150.922369] LustreError: Skipped 45 previous similar messages [1767166.232482] Lustre: fir-MDT0002: haven't heard from client fir-MDT0002-lwp-OST0034_UUID (at 10.0.10.109@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9ab0f64400, cur 1586321696 expire 1586321546 last 1586321469 [1767170.734061] Lustre: fir-MDT0002: Client 9425ec05-bd98-4 (at 10.50.7.62@o2ib2) reconnecting [1767170.742511] Lustre: Skipped 91 previous similar messages [1767172.562887] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 8 seconds [1767172.573232] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 17 previous similar messages [1767184.223953] Lustre: fir-MDT0002: Connection restored to 742c4141-354a-4 (at 10.50.4.39@o2ib2) [1767184.232659] Lustre: Skipped 168 previous similar messages [1767221.184964] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.50.12.14@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1767221.202504] LustreError: Skipped 207 previous similar messages [1767222.564181] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [1767222.576262] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 7 previous similar messages [1767239.564590] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 0 seconds [1767239.574937] LNet: 20288:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 34 previous similar messages [1767243.181693] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -11 [1767244.194720] LustreError: 20702:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0002: cannot cleanup orphans: rc = -11 [1767327.245811] LustreError: 20710:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0038-osc-MDT0002: cannot cleanup orphans: rc = -11 [1767341.462169] LustreError: 20714:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST003a-osc-MDT0002: cannot cleanup orphans: rc = -11 [1767354.567475] LNetError: 20288:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds [1767354.577652] LNetError: 20288:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.109@o2ib7 (6): c: 0, oc: 0, rc: 8 [1767360.236902] Lustre: fir-MDT0002: haven't heard from client 322961a3-22bb-4 (at 10.50.2.60@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb4ccec00, cur 1586321890 expire 1586321740 last 1586321663 [1767360.256952] Lustre: Skipped 5 previous similar messages [1767428.569313] LNetError: 20288:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [1767428.579482] LNetError: 20288:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.109@o2ib7 (5): c: 0, oc: 0, rc: 8 [1767428.591908] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [1767428.604016] LNetError: 20288:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 5 previous similar messages [1767649.484963] INFO: task mdt00_008:20883 blocked for more than 120 seconds. [1767649.491926] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1767649.499935] mdt00_008 D ffff8b8bb0a0c100 0 20883 2 0x00000080 [1767649.507223] Call Trace: [1767649.509888] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1767649.516772] [] schedule+0x29/0x70 [1767649.521934] [] rwsem_down_write_failed+0x225/0x3a0 [1767649.528571] [] call_rwsem_down_write_failed+0x17/0x30 [1767649.535455] [] down_write+0x2d/0x3d [1767649.540803] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1767649.547687] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1767649.554582] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1767649.561233] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1767649.568548] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1767649.576386] [] lod_prepare_create+0x215/0x2e0 [lod] [1767649.583116] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1767649.590518] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1767649.597663] [] lod_declare_create+0x204/0x590 [lod] [1767649.604413] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1767649.612427] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1767649.619054] [] mdd_create+0x867/0x14a0 [mdd] [1767649.625185] [] mdt_reint_open+0x224f/0x3240 [mdt] [1767649.631749] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1767649.639441] [] mdt_reint_rec+0x83/0x210 [mdt] [1767649.645634] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1767649.652353] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1767649.659606] [] mdt_intent_open+0x82/0x3a0 [mdt] [1767649.665999] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1767649.673319] [] mdt_intent_policy+0x435/0xd80 [mdt] [1767649.679964] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1767649.687318] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1767649.694207] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1767649.701539] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1767649.708104] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1767649.715368] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1767649.723089] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1767649.729395] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1767649.736479] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1767649.744263] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1767649.751525] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1767649.759391] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1767649.766374] [] ? __wake_up+0x44/0x50 [1767649.771811] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1767649.778285] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1767649.785876] [] kthread+0xd1/0xe0 [1767649.790934] [] ? insert_kthread_work+0x40/0x40 [1767649.797212] [] ret_from_fork_nospec_begin+0xe/0x21 [1767649.803866] [] ? insert_kthread_work+0x40/0x40 [1767649.810143] INFO: task mdt00_010:20918 blocked for more than 120 seconds. [1767649.817110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1767649.825131] mdt00_010 D ffff8b8b93a630c0 0 20918 2 0x00000080 [1767649.832404] Call Trace: [1767649.835040] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1767649.841926] [] schedule+0x29/0x70 [1767649.847087] [] rwsem_down_write_failed+0x225/0x3a0 [1767649.853710] [] call_rwsem_down_write_failed+0x17/0x30 [1767649.860593] [] down_write+0x2d/0x3d [1767649.865952] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1767649.872837] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1767649.879723] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1767649.886363] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1767649.893681] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1767649.901539] [] lod_prepare_create+0x215/0x2e0 [lod] [1767649.908250] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1767649.915658] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1767649.922813] [] lod_declare_create+0x204/0x590 [lod] [1767649.929526] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1767649.937544] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1767649.944198] [] mdd_create+0x867/0x14a0 [mdd] [1767649.950308] [] mdt_reint_open+0x224f/0x3240 [mdt] [1767649.956861] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1767649.964550] [] mdt_reint_rec+0x83/0x210 [mdt] [1767649.970739] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1767649.977457] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1767649.984708] [] mdt_intent_open+0x82/0x3a0 [mdt] [1767649.991081] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1767649.998405] [] mdt_intent_policy+0x435/0xd80 [mdt] [1767650.005061] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1767650.012403] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1767650.019290] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1767650.026626] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1767650.033186] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1767650.040445] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1767650.048159] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1767650.054459] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1767650.061541] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1767650.069311] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1767650.076578] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1767650.084444] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1767650.091424] [] ? __wake_up+0x44/0x50 [1767650.096855] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1767650.103338] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1767650.110924] [] kthread+0xd1/0xe0 [1767650.115979] [] ? insert_kthread_work+0x40/0x40 [1767650.122269] [] ret_from_fork_nospec_begin+0xe/0x21 [1767650.128885] [] ? insert_kthread_work+0x40/0x40 [1767650.135162] INFO: task mdt00_012:20930 blocked for more than 120 seconds. [1767650.142156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1767650.150164] mdt00_012 D ffff8b8b9c3d30c0 0 20930 2 0x00000080 [1767650.157454] Call Trace: [1767650.160091] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1767650.166967] [] schedule+0x29/0x70 [1767650.172132] [] rwsem_down_write_failed+0x225/0x3a0 [1767650.178753] [] call_rwsem_down_write_failed+0x17/0x30 [1767650.185636] [] down_write+0x2d/0x3d [1767650.190978] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1767650.197862] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1767650.204753] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1767650.211414] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1767650.218733] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1767650.226576] [] lod_prepare_create+0x215/0x2e0 [lod] [1767650.233306] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1767650.240711] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1767650.247856] [] lod_declare_create+0x204/0x590 [lod] [1767650.254582] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1767650.262595] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1767650.269215] [] mdd_create+0x867/0x14a0 [mdd] [1767650.275353] [] mdt_reint_open+0x224f/0x3240 [mdt] [1767650.281901] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1767650.289571] [] mdt_reint_rec+0x83/0x210 [mdt] [1767650.295781] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1767650.302489] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1767650.309727] [] mdt_intent_open+0x82/0x3a0 [mdt] [1767650.316125] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1767650.323446] [] mdt_intent_policy+0x435/0xd80 [mdt] [1767650.330078] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1767650.337421] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1767650.344322] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1767650.351652] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1767650.358214] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1767650.365483] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1767650.373178] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1767650.379489] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1767650.386588] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1767650.394354] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1767650.401610] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1767650.409487] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1767650.416476] [] ? __wake_up+0x44/0x50 [1767650.421907] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1767650.428396] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1767650.435971] [] kthread+0xd1/0xe0 [1767650.441034] [] ? insert_kthread_work+0x40/0x40 [1767650.447324] [] ret_from_fork_nospec_begin+0xe/0x21 [1767650.453944] [] ? insert_kthread_work+0x40/0x40 [1767650.460218] INFO: task mdt01_017:20973 blocked for more than 120 seconds. [1767650.467192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1767650.475196] mdt01_017 D ffff8b8bbef98000 0 20973 2 0x00000080 [1767650.482481] Call Trace: [1767650.485114] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1767650.491995] [] schedule+0x29/0x70 [1767650.497160] [] rwsem_down_write_failed+0x225/0x3a0 [1767650.503783] [] call_rwsem_down_write_failed+0x17/0x30 [1767650.510663] [] down_write+0x2d/0x3d [1767650.516008] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1767650.522889] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1767650.529778] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1767650.536419] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1767650.543757] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1767650.551593] [] lod_prepare_create+0x215/0x2e0 [lod] [1767650.558326] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1767650.565728] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1767650.572873] [] lod_declare_create+0x204/0x590 [lod] [1767650.579602] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1767650.587611] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1767650.594236] [] mdd_create+0x867/0x14a0 [mdd] [1767650.600366] [] mdt_reint_open+0x224f/0x3240 [mdt] [1767650.606918] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1767650.614590] [] mdt_reint_rec+0x83/0x210 [mdt] [1767650.620797] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1767650.627510] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1767650.634748] [] mdt_intent_open+0x82/0x3a0 [mdt] [1767650.641142] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1767650.648477] [] mdt_intent_policy+0x435/0xd80 [mdt] [1767650.655107] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1767650.662460] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1767650.669341] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1767650.676660] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1767650.683235] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1767650.690497] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1767650.698205] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1767650.704511] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1767650.711591] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1767650.719353] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1767650.726611] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1767650.734472] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1767650.741453] [] ? __wake_up+0x44/0x50 [1767650.746894] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1767650.753371] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1767650.760961] [] kthread+0xd1/0xe0 [1767650.766022] [] ? insert_kthread_work+0x40/0x40 [1767650.772293] [] ret_from_fork_nospec_begin+0xe/0x21 [1767650.778926] [] ? insert_kthread_work+0x40/0x40 [1767676.027528] LustreError: 20698:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0032-osc-MDT0002: cannot cleanup orphans: rc = -11 [1767734.800004] LNet: Service thread pid 20918 was inactive for 218.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1767734.817120] Pid: 20918, comm: mdt00_010 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1767734.827470] Call Trace: [1767734.830117] [] osp_precreate_reserve+0x2e8/0x800 [osp] [1767734.837126] [] osp_declare_create+0x199/0x5f0 [osp] [1767734.843880] [] lod_sub_declare_create+0xdf/0x210 [lod] [1767734.850901] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [1767734.858167] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [1767734.865624] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [1767734.872631] [] lod_prepare_create+0x215/0x2e0 [lod] [1767734.879394] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1767734.886826] [] lod_declare_create+0x204/0x590 [lod] [1767734.893581] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1767734.901642] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1767734.908307] [] mdd_create+0x867/0x14a0 [mdd] [1767734.914437] [] mdt_reint_open+0x224f/0x3240 [mdt] [1767734.921030] [] mdt_reint_rec+0x83/0x210 [mdt] [1767734.927255] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1767734.934004] [] mdt_intent_open+0x82/0x3a0 [mdt] [1767734.940406] [] mdt_intent_policy+0x435/0xd80 [mdt] [1767734.947071] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1767734.954035] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1767734.961317] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1767734.967683] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1767734.974804] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1767734.982717] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1767734.989225] [] kthread+0xd1/0xe0 [1767734.994329] [] ret_from_fork_nospec_begin+0xe/0x21 [1767735.000977] [] 0xffffffffffffffff [1767735.006201] LustreError: dumping log to /tmp/lustre-log.1586322264.20918 [1767744.560251] Lustre: 20357:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586321518/real 1586321518] req@ffff8b9f6b2de780 x1661556190515840/t0(0) o400->fir-OST003a-osc-MDT0002@10.0.10.109@o2ib7:28/4 lens 224/224 e 0 to 1 dl 1586322274 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [1767744.588798] Lustre: 20357:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 21 previous similar messages [1767776.262659] Lustre: fir-MDT0002: haven't heard from client 605a3fcc-3131-4 (at 10.50.2.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb536b400, cur 1586322306 expire 1586322156 last 1586322079 [1767805.457799] LNet: Service thread pid 20506 was inactive for 200.41s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1767805.474918] Pid: 20506, comm: mdt00_002 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1767805.485267] Call Trace: [1767805.487911] [] osp_precreate_reserve+0x2e8/0x800 [osp] [1767805.494939] [] osp_declare_create+0x199/0x5f0 [osp] [1767805.501678] [] lod_sub_declare_create+0xdf/0x210 [lod] [1767805.508695] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [1767805.515954] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [1767805.523397] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [1767805.530391] [] lod_prepare_create+0x215/0x2e0 [lod] [1767805.537143] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1767805.544570] [] lod_declare_create+0x204/0x590 [lod] [1767805.551322] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1767805.559367] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1767805.566029] [] mdd_create+0x867/0x14a0 [mdd] [1767805.572161] [] mdt_reint_open+0x224f/0x3240 [mdt] [1767805.578761] [] mdt_reint_rec+0x83/0x210 [mdt] [1767805.584984] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1767805.591759] [] mdt_intent_open+0x82/0x3a0 [mdt] [1767805.598160] [] mdt_intent_policy+0x435/0xd80 [mdt] [1767805.604846] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1767805.611792] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1767805.619088] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1767805.625425] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1767805.632556] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1767805.640442] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1767805.646945] [] kthread+0xd1/0xe0 [1767805.652052] [] ret_from_fork_nospec_begin+0xe/0x21 [1767805.658696] [] 0xffffffffffffffff [1767805.663905] LustreError: dumping log to /tmp/lustre-log.1586322335.20506 [1767809.745235] LNet: Service thread pid 20918 completed after 293.27s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1767809.761576] LNet: Skipped 1 previous similar message [1768400.764396] Lustre: fir-OST003a-osc-MDT0002: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1768400.774394] Lustre: Skipped 76 previous similar messages [1768553.853623] Lustre: fir-MDT0002: Connection restored to c68ee752-f14f-4 (at 10.50.1.13@o2ib2) [1768553.862327] Lustre: Skipped 5 previous similar messages [1768607.268189] Lustre: fir-MDT0002: haven't heard from client c68ee752-f14f-4 (at 10.50.1.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bababbbfc00, cur 1586323137 expire 1586322987 last 1586322910 [1768683.279000] Lustre: fir-MDT0002: haven't heard from client 844e77f6-3f22-4 (at 10.50.2.31@o2ib2) in 163 seconds. I think it's dead, and I am evicting it. exp ffff8baba8fb5c00, cur 1586323213 expire 1586323063 last 1586323050 [1768917.535900] Lustre: fir-MDT0002: Connection restored to 89cdc238-a9fc-4 (at 10.50.2.49@o2ib2) [1768917.544597] Lustre: Skipped 1 previous similar message [1768992.277045] Lustre: fir-MDT0002: haven't heard from client 89cdc238-a9fc-4 (at 10.50.2.49@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb019d000, cur 1586323522 expire 1586323372 last 1586323295 [1769599.871594] Lustre: fir-MDT0002: Connection restored to 21098921-f7b5-4 (at 10.50.4.6@o2ib2) [1770942.430367] Lustre: fir-MDT0002: Connection restored to 522bdb54-ddbf-4 (at 10.50.2.58@o2ib2) [1770978.403261] LNet: Service thread pid 41530 was inactive for 200.34s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1770978.420371] Pid: 41530, comm: mdt02_056 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1770978.430718] Call Trace: [1770978.433359] [] call_rwsem_down_write_failed+0x17/0x30 [1770978.440290] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1770978.447206] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1770978.454129] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [1770978.462166] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [1770978.469607] [] mdd_declare_layout_change+0x62/0x120 [mdd] [1770978.476875] [] mdd_layout_change+0xb46/0x16a0 [mdd] [1770978.483617] [] mdt_layout_change+0x2df/0x480 [mdt] [1770978.490273] [] mdt_intent_layout+0x8a0/0xe00 [mdt] [1770978.496942] [] mdt_intent_policy+0x435/0xd80 [mdt] [1770978.503601] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1770978.510560] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1770978.517842] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1770978.524207] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1770978.531330] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1770978.539229] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1770978.545730] [] kthread+0xd1/0xe0 [1770978.550850] [] ret_from_fork_nospec_begin+0xe/0x21 [1770978.557504] [] 0xffffffffffffffff [1770978.562711] LustreError: dumping log to /tmp/lustre-log.1586325508.41530 [1770980.172278] LNet: Service thread pid 41530 completed after 202.11s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1770994.327031] Lustre: fir-MDT0002: haven't heard from client 522bdb54-ddbf-4 (at 10.50.2.58@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babaf85dc00, cur 1586325524 expire 1586325374 last 1586325297 [1779125.292724] Lustre: fir-MDT0002: Connection restored to 48a86d34-282f-4 (at 10.50.5.38@o2ib2) [1779187.573064] Lustre: fir-MDT0002: haven't heard from client 18e71bec-a099-4 (at 10.50.5.38@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b9b85cc7c00, cur 1586333717 expire 1586333567 last 1586333490 [1780181.251017] Lustre: fir-MDT0002: Connection restored to 00c7f158-cc8b-4 (at 10.50.4.48@o2ib2) [1780248.561588] Lustre: fir-MDT0002: haven't heard from client 00c7f158-cc8b-4 (at 10.50.4.48@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb38ee400, cur 1586334778 expire 1586334628 last 1586334551 [1780754.695851] Lustre: fir-MDT0002: Connection restored to 9fe5415f-280d-4 (at 10.50.5.48@o2ib2) [1780810.572563] Lustre: fir-MDT0002: haven't heard from client 9fe5415f-280d-4 (at 10.50.5.48@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb173e000, cur 1586335340 expire 1586335190 last 1586335113 [1781899.124560] LNet: Service thread pid 20860 was inactive for 200.15s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1781899.141667] Pid: 20860, comm: mdt02_006 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1781899.152012] Call Trace: [1781899.154657] [] osp_precreate_reserve+0x2e8/0x800 [osp] [1781899.161682] [] osp_declare_create+0x199/0x5f0 [osp] [1781899.168418] [] lod_sub_declare_create+0xdf/0x210 [lod] [1781899.175435] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [1781899.182691] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [1781899.190118] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [1781899.197113] [] lod_prepare_create+0x215/0x2e0 [lod] [1781899.203870] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1781899.211301] [] lod_declare_create+0x204/0x590 [lod] [1781899.218061] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1781899.226101] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1781899.232764] [] mdd_create+0x867/0x14a0 [mdd] [1781899.238895] [] mdt_reint_open+0x224f/0x3240 [mdt] [1781899.245478] [] mdt_reint_rec+0x83/0x210 [mdt] [1781899.251706] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1781899.258463] [] mdt_intent_open+0x82/0x3a0 [mdt] [1781899.264861] [] mdt_intent_policy+0x435/0xd80 [mdt] [1781899.271532] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1781899.278477] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1781899.285786] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1781899.292135] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1781899.299265] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1781899.307155] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1781899.313656] [] kthread+0xd1/0xe0 [1781899.318742] [] ret_from_fork_nospec_begin+0xe/0x21 [1781899.325405] [] 0xffffffffffffffff [1781899.330602] LustreError: dumping log to /tmp/lustre-log.1586336428.20860 [1781939.928884] LNet: Service thread pid 20860 completed after 240.95s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1786392.547611] LNet: Service thread pid 20858 was inactive for 258.78s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1786392.564723] Pid: 20858, comm: mdt02_005 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1786392.575065] Call Trace: [1786392.577707] [] osp_precreate_reserve+0x2e8/0x800 [osp] [1786392.584724] [] osp_declare_create+0x199/0x5f0 [osp] [1786392.591458] [] lod_sub_declare_create+0xdf/0x210 [lod] [1786392.598480] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [1786392.605745] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [1786392.613207] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [1786392.620206] [] lod_prepare_create+0x215/0x2e0 [lod] [1786392.626969] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1786392.634393] [] lod_declare_create+0x204/0x590 [lod] [1786392.641127] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1786392.649181] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1786392.655827] [] mdd_create+0x867/0x14a0 [mdd] [1786392.661970] [] mdt_reint_open+0x224f/0x3240 [mdt] [1786392.668543] [] mdt_reint_rec+0x83/0x210 [mdt] [1786392.674792] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1786392.681525] [] mdt_intent_open+0x82/0x3a0 [mdt] [1786392.687925] [] mdt_intent_policy+0x435/0xd80 [mdt] [1786392.694591] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1786392.701550] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1786392.708848] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1786392.715187] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1786392.722321] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1786392.730215] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1786392.736716] [] kthread+0xd1/0xe0 [1786392.741831] [] ret_from_fork_nospec_begin+0xe/0x21 [1786392.748476] [] 0xffffffffffffffff [1786392.753686] LustreError: dumping log to /tmp/lustre-log.1586340922.20858 [1786413.357421] LNet: Service thread pid 20858 completed after 279.59s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1787235.667900] Lustre: fir-MDT0002: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1787279.740267] Lustre: fir-MDT0002: haven't heard from client 9b444a4b-8a4b-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7bb91f1000, cur 1586341809 expire 1586341659 last 1586341582 [1788260.709399] Lustre: fir-MDT0002: Connection restored to 632f4069-9f90-4 (at 10.50.2.10@o2ib2) [1788329.775301] Lustre: fir-MDT0002: haven't heard from client 632f4069-9f90-4 (at 10.50.2.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8babb2de3c00, cur 1586342859 expire 1586342709 last 1586342632 [1791072.344000] LNet: Service thread pid 41502 was inactive for 200.07s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1791072.361107] Pid: 41502, comm: mdt01_070 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1791072.371455] Call Trace: [1791072.374099] [] osp_precreate_reserve+0x2e8/0x800 [osp] [1791072.381107] [] osp_declare_create+0x199/0x5f0 [osp] [1791072.387856] [] lod_sub_declare_create+0xdf/0x210 [lod] [1791072.394861] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [1791072.402116] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [1791072.409541] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [1791072.416553] [] lod_prepare_create+0x215/0x2e0 [lod] [1791072.423287] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1791072.430729] [] lod_declare_create+0x204/0x590 [lod] [1791072.437467] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1791072.445525] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1791072.452190] [] mdd_create+0x867/0x14a0 [mdd] [1791072.458335] [] mdt_reint_open+0x224f/0x3240 [mdt] [1791072.464919] [] mdt_reint_rec+0x83/0x210 [mdt] [1791072.471162] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1791072.477908] [] mdt_intent_open+0x82/0x3a0 [mdt] [1791072.484318] [] mdt_intent_policy+0x435/0xd80 [mdt] [1791072.490975] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1791072.497919] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1791072.505200] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1791072.511551] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1791072.518683] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1791072.526572] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1791072.533072] [] kthread+0xd1/0xe0 [1791072.538175] [] ret_from_fork_nospec_begin+0xe/0x21 [1791072.544825] [] 0xffffffffffffffff [1791072.550040] LustreError: dumping log to /tmp/lustre-log.1586345601.41502 [1791097.986038] LNet: Service thread pid 41502 completed after 225.71s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1800112.395829] Lustre: fir-MDT0002: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1800139.172398] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.14.3@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1800184.061235] Lustre: fir-MDT0002: haven't heard from client f3629f90-ec37-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b786fdfe800, cur 1586354713 expire 1586354563 last 1586354486 [1800239.527029] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.14.3@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1803060.096541] LNet: Service thread pid 41461 was inactive for 200.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1803060.113650] Pid: 41461, comm: mdt00_048 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1803060.124000] Call Trace: [1803060.126642] [] osp_precreate_reserve+0x2e8/0x800 [osp] [1803060.133669] [] osp_declare_create+0x199/0x5f0 [osp] [1803060.140401] [] lod_sub_declare_create+0xdf/0x210 [lod] [1803060.147420] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [1803060.154677] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [1803060.162118] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [1803060.169114] [] lod_prepare_create+0x215/0x2e0 [lod] [1803060.175865] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1803060.183327] [] lod_declare_create+0x204/0x590 [lod] [1803060.190076] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1803060.198124] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1803060.204786] [] mdd_create+0x867/0x14a0 [mdd] [1803060.210914] [] mdt_reint_open+0x224f/0x3240 [mdt] [1803060.217509] [] mdt_reint_rec+0x83/0x210 [mdt] [1803060.223734] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1803060.230488] [] mdt_intent_open+0x82/0x3a0 [mdt] [1803060.236890] [] mdt_intent_policy+0x435/0xd80 [mdt] [1803060.243561] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1803060.250521] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1803060.257817] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1803060.264156] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1803060.271270] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1803060.279159] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1803060.285674] [] kthread+0xd1/0xe0 [1803060.290763] [] ret_from_fork_nospec_begin+0xe/0x21 [1803060.297409] [] 0xffffffffffffffff [1803060.302606] LustreError: dumping log to /tmp/lustre-log.1586357589.41461 [1803096.937756] LNet: Service thread pid 41461 completed after 237.28s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1805310.391443] LNet: Service thread pid 20879 was inactive for 200.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1805310.408554] Pid: 20879, comm: mdt03_010 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1805310.418900] Call Trace: [1805310.421543] [] osp_precreate_reserve+0x2e8/0x800 [osp] [1805310.428575] [] osp_declare_create+0x199/0x5f0 [osp] [1805310.435312] [] lod_sub_declare_create+0xdf/0x210 [lod] [1805310.442330] [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] [1805310.449586] [] lod_alloc_rr.constprop.19+0xeee/0x1490 [lod] [1805310.457012] [] lod_qos_prep_create+0x12fd/0x1890 [lod] [1805310.464007] [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] [1805310.472042] [] lod_declare_layout_change+0xb65/0x10f0 [lod] [1805310.479468] [] mdd_declare_layout_change+0x62/0x120 [mdd] [1805310.486724] [] mdd_layout_change+0xb46/0x16a0 [mdd] [1805310.493473] [] mdt_layout_change+0x2df/0x480 [mdt] [1805310.500130] [] mdt_intent_layout+0x8a0/0xe00 [mdt] [1805310.506787] [] mdt_intent_policy+0x435/0xd80 [mdt] [1805310.513459] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1805310.520394] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1805310.527688] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1805310.534034] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1805310.541167] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1805310.549063] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1805310.555596] [] kthread+0xd1/0xe0 [1805310.560688] [] ret_from_fork_nospec_begin+0xe/0x21 [1805310.567340] [] 0xffffffffffffffff [1805310.572544] LustreError: dumping log to /tmp/lustre-log.1586359839.20879 [1805316.373786] LNet: Service thread pid 20879 completed after 206.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1807618.903883] LustreError: 21031:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1586361847, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b99cdb4ba80/0x2f21cf2a721004e3 lrc: 3/1,0 mode: --/PR res: [0x2c00393f1:0x8975:0x0].0x0 bits 0x13/0x0 rrc: 170 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 21031 timeout: 0 lvb_type: 0 [1808869.460507] Lustre: 41485:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586363391/real 1586363391] req@ffff8b8bb473b180 x1661557086101504/t0(0) o104->fir-MDT0002@10.49.8.32@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1586363398 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1808869.487932] Lustre: 41485:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [1808920.273554] Lustre: fir-MDT0002: haven't heard from client 14f6955a-e3d1-4 (at 10.49.8.32@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8baba5650800, cur 1586363449 expire 1586363299 last 1586363222 [1808920.293646] LustreError: 41485:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.49.8.32@o2ib1) failed to reply to blocking AST (req@ffff8b8bb473b180 x1661557086101504 status 0 rc -5), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b94343b72c0/0x2f21cf2a33bc11df lrc: 4/0,0 mode: PR/PR res: [0x2c003947c:0x7f4d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.49.8.32@o2ib1 remote: 0x591bfc15453cdfc0 expref: 283809 pid: 21586 timeout: 1808972 lvb_type: 0 [1808920.336791] LustreError: 138-a: fir-MDT0002: A client on nid 10.49.8.32@o2ib1 was evicted due to a lock blocking callback time out: rc -5 [1810092.399780] SysRq : Trigger a crash [1810092.403517] BUG: unable to handle kernel NULL pointer dereference at (null) [1810092.411570] IP: [] sysrq_handle_crash+0x16/0x20 [1810092.417870] PGD 3726e33067 PUD 3bbf2d9067 PMD 0 [1810092.422742] Oops: 0002 [#1] SMP [1810092.426200] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lmv(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel ipmi_si dcdbas ipmi_devintf ses aesni_intel enclosure lrw gf128mul glue_helper ablk_helper sg pcspkr cryptd ipmi_msghandler ccp acpi_power_meter dm_multipath i2c_piix4 k10temp dm_mod ip_tables ext4 mbcache jbd2 mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic [1810092.499260] i2c_algo_bit drm_kms_helper syscopyarea mlx5_core(OE) sysfillrect mlxfw(OE) sysimgblt fb_sys_fops devlink ahci ttm libahci crct10dif_pclmul crct10dif_common mlx_compat(OE) mpt3sas(OE) drm tg3 libata crc32c_intel raid_class ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [last unloaded: mdc] [1810092.527725] CPU: 0 PID: 121305 Comm: bash Kdump: loaded Tainted: G OE ------------ 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 [1810092.540142] Hardware name: Dell Inc. PowerEdge R6415/07YXFK, BIOS 1.10.6 08/15/2019 [1810092.547968] task: ffff8ba8f97c1040 ti: ffff8b97fd7d8000 task.ti: ffff8b97fd7d8000 [1810092.555624] RIP: 0010:[] [] sysrq_handle_crash+0x16/0x20 [1810092.564342] RSP: 0018:ffff8b97fd7dbe58 EFLAGS: 00010246 [1810092.569826] RAX: ffffffff9fe64430 RBX: ffffffffa06e4f80 RCX: 0000000000000000 [1810092.577132] RDX: 0000000000000000 RSI: ffff8b7bbee13898 RDI: 0000000000000063 [1810092.584440] RBP: ffff8b97fd7dbe58 R08: ffffffffa09e38bc R09: ffffffffa0a6c1a7 [1810092.591746] R10: 00000000000014a8 R11: 00000000000014a7 R12: 0000000000000063 [1810092.599052] R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000 [1810092.606359] FS: 00007f48ec51f740(0000) GS:ffff8b7bbee00000(0000) knlGS:0000000000000000 [1810092.614619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1810092.620536] CR2: 0000000000000000 CR3: 000000381fff8000 CR4: 00000000003407f0 [1810092.627843] Call Trace: [1810092.630474] [] __handle_sysrq+0x10d/0x170 [1810092.636304] [] write_sysrq_trigger+0x28/0x40 [1810092.642400] [] proc_reg_write+0x40/0x80 [1810092.648058] [] vfs_write+0xc0/0x1f0 [1810092.653368] [] SyS_write+0x7f/0xf0 [1810092.658598] [] system_call_fastpath+0x22/0x27 [1810092.664774] Code: eb 9b 45 01 f4 45 39 65 34 75 e5 4c 89 ef e8 e2 f7 ff ff eb db 66 66 66 66 90 55 48 89 e5 c7 05 91 31 7e 00 01 00 00 00 0f ae f8 04 25 00 00 00 00 01 5d c3 66 66 66 66 90 55 31 c0 c7 05 0e [1810092.685470] RIP [] sysrq_handle_crash+0x16/0x20 [1810092.691850] RSP [1810092.695513] CR2: 0000000000000000