[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-1127.8.2.el7_lustre.x86_64 (jenkins@trevis-310-el7-x8664-3.trevis.whamcloud.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Mon Jun 8 13:48:45 UTC 2020 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.8.2.el7_lustre.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000dffeffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000dfff0000-0x00000000dfffffff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000011fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.5 present. [ 0.000000] DMI: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x120000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR variable ranges disabled: [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] CPU MTRRs all blank - virtualized system. [ 0.000000] e820: last_pfn = 0xdfff0 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x0009fff0-0x0009ffff] mapped at [ffffffffff200ff0] [ 0.000000] Base memory trampoline at [ffff9c8640099000] 99000 size 24576 [ 0.000000] BRK [0x30073000, 0x30073fff] PGTABLE [ 0.000000] BRK [0x30074000, 0x30074fff] PGTABLE [ 0.000000] BRK [0x30075000, 0x30075fff] PGTABLE [ 0.000000] BRK [0x30076000, 0x30076fff] PGTABLE [ 0.000000] BRK [0x30077000, 0x30077fff] PGTABLE [ 0.000000] RAMDISK: [mem 0x34e25000-0x3670afff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000e0000 00024 (v02 VBOX ) [ 0.000000] ACPI: XSDT 00000000dfff0030 0003C (v01 VBOX VBOXXSDT 00000001 ASL 00000061) [ 0.000000] ACPI: FACP 00000000dfff00f0 000F4 (v04 VBOX VBOXFACP 00000001 ASL 00000061) [ 0.000000] ACPI: DSDT 00000000dfff0480 02325 (v02 VBOX VBOXBIOS 00000002 INTL 20100528) [ 0.000000] ACPI: FACS 00000000dfff0200 00040 [ 0.000000] ACPI: APIC 00000000dfff0240 0006C (v02 VBOX VBOXAPIC 00000001 ASL 00000061) [ 0.000000] ACPI: SSDT 00000000dfff02b0 001CC (v01 VBOX VBOXCPUT 00000002 INTL 20100528) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000011fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x11ffd5000-0x11fffbfff] [ 0.000000] Reserving 161MB of memory at 576MB for crashkernel (System RAM: 4095MB) [ 0.000000] kvm-clock: cpu 0, msr 1:1ff84001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 8692633707 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x11fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xdffeffff] [ 0.000000] node 0: [mem 0x100000000-0x11fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x11fffffff] [ 0.000000] On node 0 totalpages: 1048462 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 14272 pages used for memmap [ 0.000000] DMA32 zone: 913392 pages, LIFO batch:31 [ 0.000000] Normal zone: 2048 pages used for memmap [ 0.000000] Normal zone: 131072 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x4008 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 4, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xdfff0000-0xdfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xe0000000-0xfebfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfedfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xe0000000-0xfebfffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s118784 r8192 d28672 u524288 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1032057 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.8.2.el7_lustre.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3851512k/4718592k available (7784k kernel code, 524744k absent, 342336k reserved, 5957k data, 1980k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2591.984 MHz processor [ 0.075763] Calibrating delay loop (skipped) preset value.. 5183.96 BogoMIPS (lpj=2591984) [ 0.075766] pid_max: default: 32768 minimum: 301 [ 0.075786] Security Framework initialized [ 0.075789] SELinux: Initializing. [ 0.075795] SELinux: Starting in permissive mode [ 0.075796] Yama: becoming mindful. [ 0.076323] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.077445] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.077654] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.077657] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.077767] Initializing cgroup subsys memory [ 0.077770] Initializing cgroup subsys devices [ 0.077771] Initializing cgroup subsys freezer [ 0.077772] Initializing cgroup subsys net_cls [ 0.077773] Initializing cgroup subsys blkio [ 0.077774] Initializing cgroup subsys perf_event [ 0.077775] Initializing cgroup subsys hugetlb [ 0.077776] Initializing cgroup subsys pids [ 0.077777] Initializing cgroup subsys net_prio [ 0.078884] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.078885] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0 [ 0.078886] tlb_flushall_shift: 6 [ 0.078893] FEATURE SPEC_CTRL Not Present [ 0.078893] FEATURE IBPB_SUPPORT Not Present [ 0.078895] Spectre V1 : Mitigation: Load fences, usercopy/swapgs barriers and __user pointer sanitization [ 0.078898] Spectre V2 : retpoline selected on command line. [ 0.079228] Spectre V2 : Vulnerable: Retpoline without IBPB [ 0.079229] Speculative Store Bypass: Vulnerable [ 0.079278] MDS: Mitigation: Clear CPU buffers [ 0.079895] Freeing SMP alternatives: 28k freed [ 0.087531] ACPI: Core revision 20130517 [ 0.088074] ACPI: All ACPI Tables successfully acquired [ 0.088158] ftrace: allocating 29619 entries in 116 pages [ 0.132652] Enabling x2apic [ 0.132658] Enabled x2apic [ 0.132834] Switched APIC routing to physical x2apic. [ 0.133914] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.133916] smpboot: CPU0: Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz (fam: 06, model: 9e, stepping: 0d) [ 0.230456] APIC calibration not consistent with PM-Timer: 103ms instead of 100ms [ 0.230459] APIC delta adjusted to PM-Timer: 6367800 (6561919) [ 0.230559] Performance Events: unsupported p6 CPU model 158 no PMU driver, software events only. [ 0.230578] KVM setup paravirtual spinlock [ 0.231672] kvm-clock: cpu 1, msr 1:1ff84041, secondary cpu clock [ 0.235026] kvm-clock: cpu 2, msr 1:1ff84081, secondary cpu clock [ 0.231426] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.238381] kvm-clock: cpu 3, msr 1:1ff840c1, secondary cpu clock [ 0.241507] Brought up 4 CPUs [ 0.241509] smpboot: Max logical packages: 1 [ 0.241510] smpboot: Total of 4 processors activated (20735.87 BogoMIPS) [ 0.242347] devtmpfs: initialized [ 0.242723] x86/mm: Memory block size: 128MB [ 0.244402] EVM: security.selinux [ 0.244403] EVM: security.ima [ 0.244404] EVM: security.capability [ 0.245530] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.245533] pinctrl core: initialized pinctrl subsystem [ 0.245568] RTC time: 19:10:43, date: 07/12/21 [ 0.245645] NET: Registered protocol family 16 [ 0.245749] cpuidle: using governor haltpoll [ 0.245790] ACPI: bus type PCI registered [ 0.245792] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.245861] PCI: Using configuration type 1 for base access [ 0.250772] ACPI: Added _OSI(Module Device) [ 0.250774] ACPI: Added _OSI(Processor Device) [ 0.250775] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250776] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250777] ACPI: Added _OSI(Linux-Dell-Video) [ 0.251068] ACPI: EC: Look up EC in DSDT [ 0.251325] ACPI: Executed 1 blocks of module-level executable AML code [ 0.252847] ACPI: Interpreter enabled [ 0.252853] ACPI: (supports S0 S5) [ 0.252854] ACPI: Using IOAPIC for interrupt routing [ 0.252961] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253022] ACPI: Enabled 2 GPEs in block 00 to 07 [ 0.255642] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255646] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.255860] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.255866] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.256073] PCI host bridge to bus 0000:00 [ 0.256080] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256082] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.256083] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.256085] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfdffffff window] [ 0.256088] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.256153] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.256533] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.256997] pci 0000:00:01.1: [8086:7111] type 00 class 0x01018a [ 0.257213] pci 0000:00:01.1: reg 0x20: [io 0xd000-0xd00f] [ 0.257316] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.257317] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.257318] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.257320] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.257475] pci 0000:00:02.0: [15ad:0405] type 00 class 0x030000 [ 0.260405] pci 0000:00:02.0: reg 0x10: [io 0xd010-0xd01f] [ 0.262977] pci 0000:00:02.0: reg 0x14: [mem 0xe0000000-0xe3ffffff pref] [ 0.265435] pci 0000:00:02.0: reg 0x18: [mem 0xf0000000-0xf01fffff] [ 0.275109] pci 0000:00:03.0: [8086:100e] type 00 class 0x020000 [ 0.277949] pci 0000:00:03.0: reg 0x10: [mem 0xf0200000-0xf021ffff] [ 0.283255] pci 0000:00:03.0: reg 0x18: [io 0xd020-0xd027] [ 0.290341] pci 0000:00:04.0: [80ee:cafe] type 00 class 0x088000 [ 0.291870] pci 0000:00:04.0: reg 0x10: [io 0xd040-0xd05f] [ 0.293327] pci 0000:00:04.0: reg 0x14: [mem 0xf0400000-0xf07fffff] [ 0.294821] pci 0000:00:04.0: reg 0x18: [mem 0xf0800000-0xf0803fff pref] [ 0.300771] pci 0000:00:05.0: [8086:2415] type 00 class 0x040100 [ 0.300834] pci 0000:00:05.0: reg 0x10: [io 0xd100-0xd1ff] [ 0.300876] pci 0000:00:05.0: reg 0x14: [io 0xd200-0xd23f] [ 0.301285] pci 0000:00:06.0: [106b:003f] type 00 class 0x0c0310 [ 0.302669] pci 0000:00:06.0: reg 0x10: [mem 0xf0804000-0xf0804fff] [ 0.314216] pci 0000:00:07.0: [8086:7113] type 00 class 0x068000 [ 0.314556] pci 0000:00:07.0: quirk: [io 0x4000-0x403f] claimed by PIIX4 ACPI [ 0.314565] pci 0000:00:07.0: quirk: [io 0x4100-0x410f] claimed by PIIX4 SMB [ 0.314772] pci 0000:00:08.0: [8086:100e] type 00 class 0x020000 [ 0.317775] pci 0000:00:08.0: reg 0x10: [mem 0xf0820000-0xf083ffff] [ 0.321459] pci 0000:00:08.0: reg 0x18: [io 0xd240-0xd247] [ 0.327661] pci 0000:00:0d.0: [8086:2829] type 00 class 0x010601 [ 0.329796] pci 0000:00:0d.0: reg 0x10: [io 0xd248-0xd24f] [ 0.334408] pci 0000:00:0d.0: reg 0x14: [io 0xd250-0xd253] [ 0.336596] pci 0000:00:0d.0: reg 0x18: [io 0xd258-0xd25f] [ 0.338447] pci 0000:00:0d.0: reg 0x1c: [io 0xd260-0xd263] [ 0.340112] pci 0000:00:0d.0: reg 0x20: [io 0xd270-0xd27f] [ 0.341622] pci 0000:00:0d.0: reg 0x24: [mem 0xf0840000-0xf0841fff] [ 0.344379] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 9 10 *11) [ 0.344563] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 9 *10 11) [ 0.344639] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 *9 10 11) [ 0.344708] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 9 10 *11) [ 0.345060] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.345063] vgaarb: loaded [ 0.345065] vgaarb: bridge control possible 0000:00:02.0 [ 0.345286] SCSI subsystem initialized [ 0.345317] ACPI: bus type USB registered [ 0.345335] usbcore: registered new interface driver usbfs [ 0.345341] usbcore: registered new interface driver hub [ 0.345369] usbcore: registered new device driver usb [ 0.345612] EDAC MC: Ver: 3.0.0 [ 0.345960] PCI: Using ACPI for IRQ routing [ 0.345963] PCI: pci_cache_line_size set to 64 bytes [ 0.346175] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.346180] e820: reserve RAM buffer [mem 0xdfff0000-0xdfffffff] [ 0.346311] NetLabel: Initializing [ 0.346312] NetLabel: domain hash size = 128 [ 0.346313] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.346327] NetLabel: unlabeled traffic allowed by default [ 0.346439] amd_nb: Cannot enumerate AMD northbridges [ 0.346529] Switched to clocksource kvm-clock [ 0.351313] pnp: PnP ACPI init [ 0.351324] ACPI: bus type PNP registered [ 0.351386] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.351442] pnp 00:01: Plug and Play ACPI device, IDs PNP0f03 (active) [ 0.351824] pnp: PnP ACPI: found 2 devices [ 0.351826] ACPI: bus type PNP unregistered [ 0.359704] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.359707] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.359708] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.359710] pci_bus 0000:00: resource 7 [mem 0xe0000000-0xfdffffff window] [ 0.359730] NET: Registered protocol family 2 [ 0.360199] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.360248] TCP bind hash table entries: 32768 (order: 7, 524288 bytes) [ 0.360281] TCP: Hash tables configured (established 32768 bind 32768) [ 0.360291] TCP: reno registered [ 0.360295] UDP hash table entries: 2048 (order: 4, 65536 bytes) [ 0.360303] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes) [ 0.360331] NET: Registered protocol family 1 [ 0.360344] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.360372] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.360393] pci 0000:00:02.0: Boot video device [ 0.361170] PCI: CLS 0 bytes, default 64 [ 0.361200] Unpacking initramfs... [ 0.656963] Freeing initrd memory: 25496k freed [ 0.659022] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 0.659027] software IO TLB [mem 0xdbff0000-0xdfff0000] (64MB) mapped at [ffff9c871bff0000-ffff9c871ffeffff] [ 0.659107] RAPL PMU: API unit is 2^-32 Joules, 5 fixed counters, 10737418240 ms ovfl timer [ 0.659108] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 0.659119] RAPL PMU: hw unit of domain package 2^-0 Joules [ 0.659120] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 0.659121] RAPL PMU: hw unit of domain pp1-gpu 2^-0 Joules [ 0.659122] RAPL PMU: hw unit of domain psys 2^-0 Joules [ 0.659260] platform rtc_cmos: registered platform RTC device (no PNP device found) [ 0.660221] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.660287] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.660555] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 0.660567] Initialise system trusted keyring [ 0.660580] audit: initializing netlink socket (disabled) [ 0.660588] type=2000 audit(1626117054.084:1): initialized [ 0.681357] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.682575] zpool: loaded [ 0.682578] zbud: loaded [ 0.682821] VFS: Disk quotas dquot_6.6.0 [ 0.682853] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.683113] Key type big_key registered [ 0.683115] SELinux: Registering netfilter hooks [ 1.186313] NET: Registered protocol family 38 [ 1.186324] Key type asymmetric registered [ 1.186327] Asymmetric key parser 'x509' registered [ 1.186374] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 1.186427] io scheduler noop registered [ 1.186429] io scheduler deadline registered (default) [ 1.186448] io scheduler cfq registered [ 1.186450] io scheduler mq-deadline registered [ 1.186453] io scheduler kyber registered [ 1.186537] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 1.186541] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.186550] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 1.186637] intel_idle: does not run on family 6 model 158 [ 1.186720] ACPI: AC Adapter [AC] (on-line) [ 1.186759] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 1.186762] ACPI: Power Button [PWRF] [ 1.186803] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1 [ 1.186805] ACPI: Sleep Button [SLPF] [ 1.186918] GHES: HEST is not enabled! [ 1.187425] ACPI: Battery Slot [BAT0] (battery present) [ 1.187546] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 1.188094] Non-volatile memory driver v1.3 [ 1.188190] Linux agpgart interface v0.103 [ 1.188317] crash memory driver: version 1.1 [ 1.188370] rdac: device handler registered [ 1.188409] hp_sw: device handler registered [ 1.188412] emc: device handler registered [ 1.188450] alua: device handler registered [ 1.188512] libphy: Fixed MDIO Bus: probed [ 1.188573] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.188577] ehci-pci: EHCI PCI platform driver [ 1.188585] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 1.188588] ohci-pci: OHCI PCI platform driver [ 1.188971] ohci-pci 0000:00:06.0: OHCI PCI host controller [ 1.189021] ohci-pci 0000:00:06.0: new USB bus registered, assigned bus number 1 [ 1.189068] ohci-pci 0000:00:06.0: irq 22, io mem 0xf0804000 [ 1.291729] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 3.10 [ 1.291732] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.291734] usb usb1: Product: OHCI PCI host controller [ 1.291736] usb usb1: Manufacturer: Linux 3.10.0-1127.8.2.el7_lustre.x86_64 ohci_hcd [ 1.291737] usb usb1: SerialNumber: 0000:00:06.0 [ 1.291960] hub 1-0:1.0: USB hub found [ 1.291972] hub 1-0:1.0: 12 ports detected [ 1.292272] uhci_hcd: USB Universal Host Controller Interface driver [ 1.292323] usbcore: registered new interface driver usbserial_generic [ 1.292329] usbserial: USB Serial support registered for generic [ 1.292364] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12 [ 1.292791] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.292796] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.292959] mousedev: PS/2 mouse device common for all mice [ 1.293262] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 [ 1.294354] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0 [ 1.294402] rtc_cmos rtc_cmos: alarms up to one day, 114 bytes nvram [ 1.294511] hidraw: raw HID events driver (C) Jiri Kosina [ 1.294599] usbcore: registered new interface driver usbhid [ 1.294601] usbhid: USB HID core driver [ 1.294649] drop_monitor: Initializing network drop monitor service [ 1.294700] TCP: cubic registered [ 1.294703] Initializing XFRM netlink socket [ 1.294791] NET: Registered protocol family 10 [ 1.295023] NET: Registered protocol family 17 [ 1.295029] mpls_gso: MPLS GSO support [ 1.295216] mce: Using 0 MCE banks [ 1.295519] PM: Hibernation image not present or could not be loaded. [ 1.295522] Loading compiled-in X.509 certificates [ 1.295545] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 1.295572] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 1.295873] Loaded X.509 cert 'CentOS Linux kernel signing key: 947e7f4116023006e4f4088f567b44a2463482b4' [ 1.295889] registered taskstats version 1 [ 1.298961] Key type trusted registered [ 1.301208] Key type encrypted registered [ 1.301226] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 1.301438] Magic number: 13:539:192 [ 1.301579] rtc_cmos rtc_cmos: setting system clock to 2021-07-12 19:10:45 UTC (1626117045) [ 1.302522] Freeing unused kernel memory: 1980k freed [ 1.302736] Write protecting the kernel read-only data: 12288k [ 1.303636] Freeing unused kernel memory: 396k freed [ 1.304666] Freeing unused kernel memory: 540k freed [ 1.307472] random: systemd: uninitialized urandom read (16 bytes read) [ 1.307759] random: systemd: uninitialized urandom read (16 bytes read) [ 1.307769] random: systemd: uninitialized urandom read (16 bytes read) [ 1.308436] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 1.308454] systemd[1]: Detected virtualization kvm. [ 1.308462] systemd[1]: Detected architecture x86-64. [ 1.308464] systemd[1]: Running in initial RAM disk. [ 1.308492] systemd[1]: Set hostname to <mds1>. [ 1.340818] random: systemd: uninitialized urandom read (16 bytes read) [ 1.340833] random: systemd: uninitialized urandom read (16 bytes read) [ 1.340853] random: systemd: uninitialized urandom read (16 bytes read) [ 1.340885] random: systemd: uninitialized urandom read (16 bytes read) [ 1.341375] random: systemd: uninitialized urandom read (16 bytes read) [ 1.341472] random: systemd: uninitialized urandom read (16 bytes read) [ 1.341759] random: systemd: uninitialized urandom read (16 bytes read) [ 1.344933] systemd[1]: Reached target Local File Systems. [ 1.344988] systemd[1]: Reached target Timers. [ 1.344997] systemd[1]: Reached target Swap. [ 1.345243] systemd[1]: Created slice Root Slice. [ 1.345329] systemd[1]: Created slice System Slice. [ 1.345404] systemd[1]: Listening on udev Control Socket. [ 1.345411] systemd[1]: Reached target Slices. [ 1.345437] systemd[1]: Listening on udev Kernel Socket. [ 1.345487] systemd[1]: Listening on Journal Socket. [ 1.346152] systemd[1]: Starting Setup Virtual Console... [ 1.346190] systemd[1]: Reached target Sockets. [ 1.346919] systemd[1]: Starting dracut cmdline hook... [ 1.347713] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 1.348422] systemd[1]: Starting Journal Service... [ 1.349569] systemd[1]: Starting Apply Kernel Variables... [ 1.350160] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 1.351531] systemd[1]: Starting Create Static Device Nodes in /dev... [ 1.352057] systemd[1]: Started Apply Kernel Variables. [ 1.355308] systemd[1]: Started Create Static Device Nodes in /dev. [ 1.366936] systemd[1]: Started Journal Service. [ 1.486912] device-mapper: uevent: version 1.0.3 [ 1.486988] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 1.494876] psmouse serio1: alps: Unknown ALPS touchpad: E7=10 00 64, EC=10 00 64 [ 1.495798] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 [ 1.567062] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI [ 1.567066] e1000: Copyright (c) 1999-2006 Intel Corporation. [ 1.587465] libata version 3.00 loaded. [ 1.664513] tsc: Refined TSC clocksource calibration: 2591.562 MHz [ 1.906494] e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 08:00:27:4d:54:1b [ 1.906503] e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection [ 3.314904] e1000 0000:00:08.0 eth1: (PCI:33MHz:32-bit) 08:00:27:4c:a5:06 [ 3.314911] e1000 0000:00:08.0 eth1: Intel(R) PRO/1000 Network Connection [ 3.315032] ata_piix 0000:00:01.1: version 2.13 [ 3.315811] scsi host0: ata_piix [ 3.316042] scsi host1: ata_piix [ 3.316087] ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0xd000 irq 14 [ 3.316089] ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0xd008 irq 15 [ 3.316120] ahci 0000:00:0d.0: version 3.0 [ 3.316692] ahci 0000:00:0d.0: SSS flag set, parallel bus scan disabled [ 3.316844] ahci 0000:00:0d.0: AHCI 0001.0100 32 slots 2 ports 3 Gbps 0x3 impl SATA mode [ 3.316847] ahci 0000:00:0d.0: flags: 64bit ncq stag only ccc [ 3.317233] scsi host2: ahci [ 3.317331] scsi host3: ahci [ 3.317376] ata3: SATA max UDMA/133 abar m8192@0xf0840000 port 0xf0840100 irq 21 [ 3.317386] ata4: SATA max UDMA/133 abar m8192@0xf0840000 port 0xf0840180 irq 21 [ 3.318765] [drm] DMA map mode: Keeping DMA mappings. [ 3.318819] [drm] Capabilities: [ 3.318821] [drm] Cursor. [ 3.318823] [drm] Cursor bypass 2. [ 3.318824] [drm] Alpha cursor. [ 3.318825] [drm] 3D. [ 3.318827] [drm] Extended Fifo. [ 3.318828] [drm] Pitchlock. [ 3.318829] [drm] Irq mask. [ 3.318831] [drm] GMR. [ 3.318832] [drm] Traces. [ 3.318834] [drm] GMR2. [ 3.318835] [drm] Screen Object 2. [ 3.318837] [drm] Max GMR ids is 8192 [ 3.318839] [drm] Max number of GMR pages is 1048576 [ 3.318841] [drm] Max dedicated hypervisor surface memory is 458752 kiB [ 3.318843] [drm] Maximum display memory size is 65536 kiB [ 3.318845] [drm] VRAM at 0xe0000000 size is 65536 kiB [ 3.318847] [drm] MMIO at 0xf0000000 size is 2048 kiB [ 3.319069] [TTM] Zone kernel: Available graphics memory: 1939976 kiB [ 3.319072] [TTM] Initializing pool allocator [ 3.319078] [TTM] Initializing DMA pool allocator [ 3.319106] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 3.319108] [drm] No driver support for vblank timestamp query. [ 3.319458] [drm] Screen Objects Display Unit initialized [ 3.319543] [drm] width 720 [ 3.319553] [drm] height 400 [ 3.319585] [drm] bpp 32 [ 3.320040] [drm] Fifo max 0x00200000 min 0x00001000 cap 0x00000355 [ 3.320053] [drm] DX: no. [ 3.320059] [drm] Atomic: yes. [ 3.320061] [drm] SM4_1: no. [ 3.320082] [drm:vmw_host_log [vmwgfx]] *ERROR* Failed to send host log message. [ 3.321043] [drm:vmw_host_log [vmwgfx]] *ERROR* Failed to send host log message. [ 3.325567] fbcon: svgadrmfb (fb0) is primary device [ 3.343435] Console: switching to colour frame buffer device 100x37 [ 3.345430] [drm] Initialized vmwgfx 2.15.0 20180704 for 0000:00:02.0 on minor 0 [ 3.480341] ata2.00: ATAPI: VBOX CD-ROM, 1.0, max UDMA/133 [ 3.480678] ata2.00: configured for UDMA/33 [ 3.481232] scsi 1:0:0:0: CD-ROM VBOX CD-ROM 1.0 PQ: 0 ANSI: 5 [ 3.622776] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 3.622925] ata3.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133 [ 3.622928] ata3.00: 104857600 sectors, multi 128: LBA48 NCQ (depth 31/32) [ 3.622994] ata3.00: configured for UDMA/133 [ 3.623073] scsi 2:0:0:0: Direct-Access ATA VBOX HARDDISK 1.0 PQ: 0 ANSI: 5 [ 3.981801] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 3.981974] ata4.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133 [ 3.981978] ata4.00: 209715200 sectors, multi 128: LBA48 NCQ (depth 31/32) [ 3.982075] ata4.00: configured for UDMA/133 [ 3.982182] scsi 3:0:0:0: Direct-Access ATA VBOX HARDDISK 1.0 PQ: 0 ANSI: 5 [ 4.264016] sd 2:0:0:0: [sda] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB) [ 4.264041] sd 2:0:0:0: [sda] Write Protect is off [ 4.264047] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 4.264092] sd 3:0:0:0: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB) [ 4.264109] sd 3:0:0:0: [sdb] Write Protect is off [ 4.264110] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 4.264118] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.264158] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.264735] sda: sda1 sda2 [ 4.264973] sd 3:0:0:0: [sdb] Attached SCSI disk [ 4.264979] sd 2:0:0:0: [sda] Attached SCSI disk [ 4.267591] sr 1:0:0:0: [sr0] scsi3-mmc drive: 32x/32x xa/form2 tray [ 4.267593] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 4.268170] sr 1:0:0:0: Attached scsi CD-ROM sr0 [ 4.280246] random: fast init done [ 5.252840] SGI XFS with ACLs, security attributes, no debug enabled [ 5.255586] XFS (dm-0): Mounting V5 Filesystem [ 5.701679] XFS (dm-0): Starting recovery (logdev: internal) [ 5.707566] XFS (dm-0): Ending recovery (logdev: internal) [ 5.972293] systemd-journald[108]: Received SIGTERM from PID 1 (systemd). [ 5.972563] random: crng init done [ 6.033511] SELinux: Disabled at runtime. [ 6.033527] SELinux: Unregistering netfilter hooks [ 9.313184] type=1404 audit(1626117053.511:2): selinux=0 auid=4294967295 ses=4294967295 [ 10.119802] ip_tables: (C) 2000-2006 Netfilter Core Team [ 10.120856] systemd[1]: Inserted module 'ip_tables' [ 10.296265] systemd-journald[535]: Received request to flush runtime journal from PID 1 [ 10.516423] ACPI: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 10.516476] input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A03:00/LNXVIDEO:00/input/input4 [ 10.536305] vboxguest: loading out-of-tree module taints kernel. [ 10.537033] vboxguest: module verification failed: signature and/or required key missing - tainting kernel [ 10.538304] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.540133] piix4_smbus 0000:00:07.0: SMBus Host Controller at 0x4100, revision 0 [ 10.541806] sr 1:0:0:0: Attached scsi generic sg0 type 5 [ 10.541854] sd 2:0:0:0: Attached scsi generic sg1 type 0 [ 10.541894] sd 3:0:0:0: Attached scsi generic sg2 type 0 [ 10.552199] cryptd: max_cpu_qlen set to 1000 [ 10.553736] vgdrvHeartbeatInit: Setting up heartbeat to trigger every 2000 milliseconds [ 10.554538] input: Unspecified device as /devices/pci0000:00/0000:00:04.0/input/input6 [ 10.561395] vboxguest: Successfully loaded version 6.1.10 [ 10.561413] vboxguest: misc device minor 58, IRQ 20, I/O port d040, MMIO at 00000000f0400000 (size 0x400000) [ 10.561414] vboxguest: Successfully loaded version 6.1.10 (interface 0x00010004) [ 10.568670] AVX2 version of gcm_enc/dec engaged. [ 10.568673] AES CTR mode by8 optimization enabled [ 10.590970] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 10.591214] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 10.612384] Adding 5242876k swap on /dev/mapper/centos-swap. Priority:-2 extents:1 across:5242876k FS [ 10.641450] ppdev: user-space parallel port driver [ 10.731721] XFS (sda1): Mounting V5 Filesystem [ 10.749593] XFS (sda1): Starting recovery (logdev: internal) [ 10.754416] XFS (sda1): Ending recovery (logdev: internal) [ 11.032628] snd_intel8x0 0000:00:05.0: white list rate for 1028:0177 is 48000 [ 13.609702] floppy0: no floppy controllers found [ 13.609726] work still pending [ 13.677525] type=1305 audit(1626117057.875:3): audit_pid=895 old=0 auid=4294967295 ses=4294967295 res=1 [ 13.689916] RPC: Registered named UNIX socket transport module. [ 13.689918] RPC: Registered udp transport module. [ 13.689919] RPC: Registered tcp transport module. [ 13.689920] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 14.627356] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 14.627627] IPv6: ADDRCONF(NETDEV_UP): enp0s3: link is not ready [ 14.627835] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s3: link becomes ready [ 16.941541] IPv6: ADDRCONF(NETDEV_UP): enp0s8: link is not ready [ 16.942069] e1000: enp0s8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 16.942313] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s8: link becomes ready [ 18.868664] 19:11:03.067124 main VBoxService 6.1.10 r138449 (verbosity: 0) linux.amd64 (Jun 4 2020 18:43:22) release log 19:11:03.067127 main Log opened 2021-07-12T19:11:03.067090000Z [ 18.868707] 19:11:03.067201 main OS Product: Linux [ 18.868722] 19:11:03.067220 main OS Release: 3.10.0-1127.8.2.el7_lustre.x86_64 [ 18.868736] 19:11:03.067235 main OS Version: #1 SMP Mon Jun 8 13:48:45 UTC 2020 [ 18.868754] 19:11:03.067249 main Executable: /opt/VBoxGuestAdditions-6.1.10/sbin/VBoxService 19:11:03.067250 main Process ID: 1320 19:11:03.067250 main Package type: LINUX_64BITS_GENERIC [ 18.869448] 19:11:03.067934 main 6.1.10 r138449 started. Verbose level = 0 [ 18.870237] 19:11:03.068701 main vbglR3GuestCtrlDetectPeekGetCancelSupport: Supported (#1) [ 21.102991] FS-Cache: Loaded [ 21.129563] FS-Cache: Netfs 'nfs' registered for caching [ 21.133097] Key type dns_resolver registered [ 21.158236] NFS: Registering the id_resolver key type [ 21.158248] Key type id_resolver registered [ 21.158249] Key type id_legacy registered [ 128.588105] LNet: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.590498] alg: No test for adler32 (adler32-zlib) [ 129.349366] Lustre: DEBUG MARKER: mds1: executing set_hostid [ 133.421803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 137.790691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.255729] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.613036] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.215993] Lustre: Lustre: Build Version: 2.12.7_RC1 [ 149.297701] LNet: Added LNI 10.0.2.4@tcp [8/256/0/180] [ 149.297707] LNet: Added LNI 10.0.2.104@tcp1 [8/256/0/180] [ 149.298010] LNet: Accept secure, port 988 [ 149.628948] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.657696] Lustre: MGS: Connection restored to MGC10.0.2.4@tcp_0 (at 0@lo) [ 150.413008] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 152.941691] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.024916] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.091433] Lustre: Setting parameter lustrefs-MDT0000.mdt.identity_upcall in log lustrefs-MDT0000 [ 153.120880] Lustre: ctl-lustrefs-MDT0000: No data found on store. Initialize space [ 153.148197] Lustre: lustrefs-MDT0000: new disk, initializing [ 153.249732] Lustre: lustrefs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.260644] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.938960] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 157.616845] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.702428] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.746314] Lustre: Setting parameter lustrefs-MDT0001.mdt.identity_upcall in log lustrefs-MDT0001 [ 157.764148] Lustre: srv-lustrefs-MDT0001: No data found on store. Initialize space [ 157.764150] Lustre: Skipped 1 previous similar message [ 157.792226] Lustre: lustrefs-MDT0001: new disk, initializing [ 157.807072] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [ 158.265070] Lustre: lustrefs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.276793] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.276822] Lustre: cli-ctl-lustrefs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.941120] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 163.212275] Lustre: MGS: Connection restored to 35f03572-7832-4520-ab76-24e2505f6d2b (at 10.0.2.5@tcp) [ 163.212278] Lustre: Skipped 3 previous similar messages [ 168.915833] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [ 168.915837] Lustre: Skipped 2 previous similar messages [ 169.063334] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.129976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7049.312263] 21:08:14.153622 control Session 0 is about to close ... [ 7049.312332] 21:08:14.153799 control Stopping all guest processes ... [ 7049.312478] 21:08:14.153864 control Closing all guest files ... [ 7049.312585] e1000: enp0s8 NIC Link is Down [ 7049.313169] 21:08:14.154615 control vbglR3GuestCtrlDetectPeekGetCancelSupport: Supported (#1) [ 7051.188427] e1000: enp0s3 NIC Link is Down [ 7051.188454] e1000 0000:00:03.0 enp0s3: Reset adapter [ 7053.294290] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 7053.322807] e1000: enp0s8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 7054.471441] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [ 7054.471477] Lustre: MGS: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [ 7054.471479] Lustre: Skipped 4 previous similar messages [ 7057.472158] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626124095/real 1626124095] req@ffff9c86d6db0000 x1705107463104640/t0(0) o13->lustrefs-OST0001-osc-MDT0000@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626124102 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 7057.472180] Lustre: lustrefs-OST0001-osc-MDT0000: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7058.482426] Lustre: 3094:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626124095/real 1626124095] req@ffff9c86d6db0900 x1705107463104704/t0(0) o13->lustrefs-OST0001-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626124102 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 7058.482553] Lustre: lustrefs-OST0001-osc-MDT0001: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7058.575675] 07:05:52.103132 timesync vgsvcTimeSyncWorker: Radical host time change: 35 858 696 000 000ns (HostNow=1 626 159 952 103 000 000 ns HostLast=1 626 124 093 407 000 000 ns) [ 7059.482618] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626124097/real 1626124097] req@ffff9c86d6db2880 x1705107463105152/t0(0) o400->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626124104 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7059.482622] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7063.367158] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [ 7063.367188] Lustre: MGS: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [ 7067.956209] Lustre: lustrefs-MDT0000: Client lustrefs-MDT0000-lwp-OST0000_UUID (at 10.0.2.5@tcp) reconnecting [ 7067.956249] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [ 7067.956251] Lustre: Skipped 2 previous similar messages [ 7068.389337] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [ 7068.389340] Lustre: Skipped 2 previous similar messages [ 7068.578648] 07:06:02.106104 timesync vgsvcTimeSyncWorker: Radical guest time change: 35 858 689 095 000ns (GuestNow=1 626 159 962 106 086 000 ns GuestLast=1 626 124 103 416 991 000 ns fSetTimeLastLoop=true ) [ 7072.774142] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [ 7072.774220] Lustre: MGS: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [ 7072.774222] Lustre: Skipped 10 previous similar messages [ 7078.563860] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [ 7078.563862] Lustre: Skipped 5 previous similar messages [ 7078.563914] Lustre: lustrefs-MDT0001: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [ 7087.007292] Lustre: lustrefs-MDD0000: changelog on [ 7220.428120] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626160106/real 1626160106] req@ffff9c86d1399200 x1705107463137728/t0(0) o13->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626160113 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 7220.428124] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 7220.428145] Lustre: lustrefs-OST0000-osc-MDT0001: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7220.428147] Lustre: Skipped 2 previous similar messages [ 7220.429924] Lustre: lustrefs-OST0000-osc-MDT0001: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [ 7225.456571] Lustre: lustrefs-MDT0001: Client lustrefs-MDT0001-lwp-OST0001_UUID (at 10.0.2.5@tcp) reconnecting [ 7226.452422] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626160111/real 1626160111] req@ffff9c86d49e0000 x1705107463138688/t0(0) o13->lustrefs-OST0001-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626160118 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 7226.452426] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7226.452460] Lustre: lustrefs-OST0001-osc-MDT0001: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7227.461073] Lustre: lustrefs-MDT0000: Client lustrefs-MDT0000-lwp-OST0000_UUID (at 10.0.2.5@tcp) reconnecting [ 7227.461076] Lustre: Skipped 3 previous similar messages [ 7250.577030] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [ 7250.577075] Lustre: MGS: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [ 7250.577077] Lustre: Skipped 7 previous similar messages [ 7254.595270] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [ 7322.053126] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [ 7322.053161] Lustre: MGS: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [ 7322.053163] Lustre: Skipped 1 previous similar message [ 7323.062410] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [ 7323.787528] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626160209/real 1626160209] req@ffff9c86d49d5580 x1705107463159168/t0(0) o13->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626160216 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 7323.787532] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7323.787551] Lustre: lustrefs-OST0000-osc-MDT0001: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7323.787552] Lustre: Skipped 1 previous similar message [ 7340.418296] Lustre: lustrefs-MDT0000: Client lustrefs-MDT0000-lwp-OST0000_UUID (at 10.0.2.5@tcp) reconnecting [ 7340.418297] Lustre: lustrefs-MDT0000: Export ffff9c86d72f5800 already connecting from 10.0.2.5@tcp [ 7731.062130] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [ 7731.062179] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [ 7731.062182] Lustre: Skipped 10 previous similar messages [ 7731.062196] Lustre: MGS: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [ 7731.062198] Lustre: Skipped 16 previous similar messages [ 7948.295785] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626160834/real 1626160834] req@ffff9c86d49d2880 x1705107463281088/t0(0) o13->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626160841 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 7948.295788] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 7948.295809] Lustre: lustrefs-OST0000-osc-MDT0001: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7948.295810] Lustre: Skipped 3 previous similar messages [ 7948.297293] Lustre: lustrefs-OST0000-osc-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [ 7948.297295] Lustre: Skipped 1 previous similar message [ 7949.141030] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [ 7949.296745] Lustre: lustrefs-OST0001-osc-MDT0000: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7949.296748] Lustre: Skipped 1 previous similar message [ 7951.145072] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [ 7999.181551] nfs: server dev not responding, timed out [18603.903486] Lustre: lustrefs-MDT0000: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [18603.903489] Lustre: Skipped 4 previous similar messages [18603.903606] Lustre: lustrefs-MDT0000: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [18603.903607] Lustre: Skipped 8 previous similar messages [18809.757026] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626171696/real 1626171696] req@ffff9c86d15a0d80 x1705107465361856/t0(0) o13->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626171703 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [18809.757029] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [18809.757049] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [18809.758189] Lustre: lustrefs-OST0000-osc-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [18810.795140] Lustre: lustrefs-OST0001-osc-MDT0000: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [18810.795143] Lustre: Skipped 1 previous similar message [18812.826035] Lustre: lustrefs-MDT0000: Client lustrefs-MDT0000-lwp-OST0001_UUID (at 10.0.2.5@tcp) reconnecting [18812.826039] Lustre: Skipped 1 previous similar message [18814.778486] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626171701/real 1626171701] req@ffff9c8758548900 x1705107465362496/t0(0) o400->lustrefs-OST0001-osc-MDT0000@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626171708 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [18814.778490] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [18814.800640] Lustre: lustrefs-OST0000-osc-MDT0001: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [18818.072961] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [19126.404258] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [19126.404274] Lustre: Skipped 4 previous similar messages [19126.404338] Lustre: lustrefs-MDT0001: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [19126.404339] Lustre: Skipped 10 previous similar messages [26906.600842] Lustre: Failing over lustrefs-MDT0000 [26906.749813] Lustre: server umount lustrefs-MDT0000 complete [26906.773123] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [26910.436140] LustreError: 11-0: lustrefs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [26910.436169] Lustre: lustrefs-MDT0000-osp-MDT0001: Connection to lustrefs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [26910.436408] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [26910.436444] LustreError: Skipped 2 previous similar messages [26911.778794] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [26911.778833] LustreError: Skipped 1 previous similar message [26915.512691] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [26915.512759] LustreError: Skipped 2 previous similar messages [26920.523418] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [26920.523482] LustreError: Skipped 4 previous similar messages [26930.549965] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [26930.549966] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [26930.549968] LustreError: Skipped 8 previous similar messages [26965.797098] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [26965.797219] LustreError: Skipped 34 previous similar messages [26975.381388] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [26975.483283] Lustre: lustrefs-MDT0001: Connection restored to 10.0.2.4@tcp (at 0@lo) [26975.483286] Lustre: Skipped 1 previous similar message [26975.545051] LustreError: 3679:0:(mgs_handler.c:282:mgs_revoke_lock()) MGS: can't take cfg lock for 0x736665727473756c/0x2 : rc = -11 [26975.546296] Lustre: lustrefs-MDT0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [26975.559403] Lustre: lustrefs-MDD0000: changelog on [26975.561119] Lustre: lustrefs-MDT0000: in recovery but waiting for the first client to connect [26977.237539] Lustre: lustrefs-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [26980.518587] LustreError: 167-0: lustrefs-MDT0000-lwp-MDT0001: This client was evicted by lustrefs-MDT0000; in progress operations using this service will fail. [26980.537250] Lustre: lustrefs-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [35005.953387] 14:51:39.459625 control Session 0 is about to close ... [35005.953655] 14:51:39.461972 control Stopping all guest processes ... [35005.953724] 14:51:39.462215 control Closing all guest files ... [35005.954456] 14:51:39.462932 control vbglR3GuestCtrlDetectPeekGetCancelSupport: Supported (#1) [35006.253956] e1000: enp0s3 NIC Link is Down [35006.450693] e1000: enp0s8 NIC Link is Down [35006.450701] e1000 0000:00:08.0 enp0s8: Reset adapter [35006.700055] 16:26:05.470156 timesync vgsvcTimeSyncWorker: Radical host time change: 5 675 255 000 000ns (HostNow=1 626 193 565 470 000 000 ns HostLast=1 626 187 890 215 000 000 ns) [35008.564779] e1000: enp0s8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [35008.565133] e1000 0000:00:08.0 enp0s8: Detected Tx Unit Hang Tx Queue <0> TDH <0> TDT <3> next_to_use <3> next_to_clean <0> buffer_info[next_to_clean] time_stamp <102119693> next_to_watch <1> jiffies <102119dec> next_to_watch.status <0> [35008.724408] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626187900/real 1626187900] req@ffff9c86d483b600 x1705107468461120/t0(0) o13->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626187907 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [35008.724432] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [35008.724567] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [35008.724569] Lustre: Skipped 1 previous similar message [35009.187060] Lustre: MGS: Client 35f03572-7832-4520-ab76-24e2505f6d2b (at 10.0.2.5@tcp) reconnecting [35009.187063] Lustre: Skipped 1 previous similar message [35009.187096] Lustre: MGS: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [35009.187097] Lustre: Skipped 8 previous similar messages [35010.332060] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [35010.573490] e1000 0000:00:08.0 enp0s8: Detected Tx Unit Hang Tx Queue <0> TDH <0> TDT <3> next_to_use <3> next_to_clean <0> buffer_info[next_to_clean] time_stamp <102119693> next_to_watch <1> jiffies <10211a5c4> next_to_watch.status <0> [35012.583726] e1000 0000:00:08.0 enp0s8: Detected Tx Unit Hang Tx Queue <0> TDH <0> TDT <3> next_to_use <3> next_to_clean <0> buffer_info[next_to_clean] time_stamp <102119693> next_to_watch <1> jiffies <10211ad9f> next_to_watch.status <0> [35014.312991] ------------[ cut here ]------------ [35014.313082] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260 [35014.313089] NETDEV WATCHDOG: enp0s8 (e1000): transmit queue 0 timed out [35014.313095] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) mbcache jbd2 libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev iosf_mbi crc32_pclmul snd_intel8x0 snd_ac97_codec ac97_bus snd_seq ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg i2c_piix4 pcspkr vboxguest(OE) snd_timer parport_pc snd parport soundcore video ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci ata_piix libata crct10dif_pclmul crct10dif_common e1000 crc32c_intel [35014.313231] serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [35014.313247] CPU: 2 PID: 0 Comm: swapper/2 Kdump: loaded Tainted: G OE ------------ 3.10.0-1127.8.2.el7_lustre.x86_64 #1 [35014.313251] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [35014.313258] Call Trace: [35014.313265] <IRQ> [<ffffffffa317ffa5>] dump_stack+0x19/0x1b [35014.313298] [<ffffffffa2a9bd18>] __warn+0xd8/0x100 [35014.313327] [<ffffffffa2a9bd9f>] warn_slowpath_fmt+0x5f/0x80 [35014.313361] [<ffffffffa30811e8>] dev_watchdog+0x248/0x260 [35014.313368] [<ffffffffa3080fa0>] ? dev_deactivate_queue.constprop.27+0x60/0x60 [35014.313376] [<ffffffffa2aac7c8>] call_timer_fn+0x38/0x110 [35014.313382] [<ffffffffa3080fa0>] ? dev_deactivate_queue.constprop.27+0x60/0x60 [35014.313391] [<ffffffffa2aaec5d>] run_timer_softirq+0x24d/0x300 [35014.313409] [<ffffffffa2aa5695>] __do_softirq+0xf5/0x280 [35014.313436] [<ffffffffa319642c>] call_softirq+0x1c/0x30 [35014.313456] [<ffffffffa2a2f715>] do_softirq+0x65/0xa0 [35014.313462] [<ffffffffa2aa5a15>] irq_exit+0x105/0x110 [35014.313470] [<ffffffffa31979c8>] smp_apic_timer_interrupt+0x48/0x60 [35014.313490] [<ffffffffa3193efa>] apic_timer_interrupt+0x16a/0x170 [35014.313497] <EOI> [<ffffffffa2ad53fe>] ? finish_task_switch+0x4e/0x1c0 [35014.313536] [<ffffffffa3187c40>] ? __cpuidle_text_start+0x8/0x8 [35014.313547] [<ffffffffa3187e8b>] ? native_safe_halt+0xb/0x20 [35014.313560] [<ffffffffa3187c5e>] default_idle+0x1e/0xc0 [35014.313573] [<ffffffffa2a37c80>] arch_cpu_idle+0x20/0xc0 [35014.313586] [<ffffffffa2b01c2a>] cpu_startup_entry+0x14a/0x1e0 [35014.313632] [<ffffffffa2a5a517>] start_secondary+0x1f7/0x270 [35014.313647] [<ffffffffa2a000d5>] start_cpu+0x5/0x14 [35014.313653] ---[ end trace 990b2a976e4cfb88 ]--- [35014.316514] e1000 0000:00:08.0 enp0s8: Reset adapter [35014.692009] Lustre: 3094:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626193566/real 1626193566] req@ffff9c86d7288d80 x1705107468461568/t0(0) o400->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626193573 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [35014.692042] Lustre: lustrefs-OST0000-osc-MDT0001: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [35015.703292] Lustre: lustrefs-OST0001-osc-MDT0001: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [35016.362016] e1000: enp0s8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [35016.706159] 16:26:15.475124 timesync vgsvcTimeSyncWorker: Radical guest time change: 5 675 266 834 000ns (GuestNow=1 626 193 575 475 116 000 ns GuestLast=1 626 187 900 208 282 000 ns fSetTimeLastLoop=true ) [35016.811899] Lustre: MGS: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [35020.765420] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1626193572/real 1626193579] req@ffff9c86d483e300 x1705107468462976/t0(0) o400->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626193579 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [35020.765459] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [35021.617319] Lustre: lustrefs-OST0001-osc-MDT0001: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [35021.617322] Lustre: Skipped 4 previous similar messages [35022.284910] Lustre: lustrefs-MDT0000: haven't heard from client lustrefs-MDT0000-lwp-OST0000_UUID (at 10.0.2.5@tcp) in 5685 seconds. I think it's dead, and I am evicting it. exp ffff9c86d72ae000, cur 1626193581 expire 1626193551 last 1626187896 [35027.693474] Lustre: MGS: Client 35f03572-7832-4520-ab76-24e2505f6d2b (at 10.0.2.5@tcp) reconnecting [35027.693477] Lustre: Skipped 3 previous similar messages [35029.699468] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [35029.699471] Lustre: Skipped 4 previous similar messages [35123.777879] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [35123.777882] Lustre: Skipped 2 previous similar messages [35123.778013] Lustre: lustrefs-MDT0001: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [35123.778014] Lustre: Skipped 1 previous similar message [35128.381695] Lustre: lustrefs-MDT0000: Export ffff9c86d1136c00 already connecting from 10.0.2.5@tcp [35314.490427] Lustre: 3092:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626193866/real 1626193866] req@ffff9c875838ec00 x1705107468520768/t0(0) o400->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626193873 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [35314.490454] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [35314.490456] Lustre: Skipped 1 previous similar message [35314.491768] Lustre: lustrefs-OST0000-osc-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [35314.491771] Lustre: Skipped 11 previous similar messages [35316.930340] Lustre: lustrefs-MDT0000: Client lustrefs-MDT0000-lwp-OST0000_UUID (at 10.0.2.5@tcp) reconnecting [35316.930343] Lustre: Skipped 11 previous similar messages [35320.512910] Lustre: lustrefs-OST0001-osc-MDT0000: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [35320.512914] Lustre: Skipped 1 previous similar message [35381.595137] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [35381.595140] Lustre: Skipped 4 previous similar messages [35381.595206] Lustre: lustrefs-MDT0001: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [35381.595207] Lustre: Skipped 8 previous similar messages [35385.609070] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [35642.537625] Lustre: lustrefs-MDT0000: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [35642.537629] Lustre: Skipped 1 previous similar message [35642.537738] Lustre: lustrefs-MDT0000: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [35642.537740] Lustre: Skipped 2 previous similar messages [35643.225758] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626194194/real 1626194194] req@ffff9c86d5041b00 x1705107468584896/t0(0) o13->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626194201 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [35643.225761] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [35643.225783] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [35643.225784] Lustre: Skipped 1 previous similar message [35643.568152] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [36683.514972] 16:54:02.282468 control Session 0 is about to close ... [36683.517711] 16:54:02.285218 control Stopping all guest processes ... [36683.523653] 16:54:02.288127 control Closing all guest files ... [36683.525209] 16:54:02.292622 control vbglR3GuestCtrlDetectPeekGetCancelSupport: Supported (#1) [36684.693103] e1000: enp0s3 NIC Link is Down [36684.693156] e1000: enp0s8 NIC Link is Down [36684.693170] e1000 0000:00:08.0 enp0s8: Reset adapter [36686.736336] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [36686.817016] e1000: enp0s8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [36686.817383] e1000 0000:00:08.0 enp0s8: Detected Tx Unit Hang Tx Queue <0> TDH <0> TDT <4> next_to_use <4> next_to_clean <0> buffer_info[next_to_clean] time_stamp <1022b31a1> next_to_watch <1> jiffies <1022b3996> next_to_watch.status <0> [36688.822110] e1000 0000:00:08.0 enp0s8: Detected Tx Unit Hang Tx Queue <0> TDH <0> TDT <4> next_to_use <4> next_to_clean <0> buffer_info[next_to_clean] time_stamp <1022b31a1> next_to_watch <1> jiffies <1022b416b> next_to_watch.status <0> [36689.668537] 12:37:26.832100 timesync vgsvcTimeSyncWorker: Radical host time change: 243 808 398 000 000ns (HostNow=1 626 439 046 832 000 000 ns HostLast=1 626 195 238 434 000 000 ns) [36690.049883] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626195246/real 1626195246] req@ffff9c86d0ecd100 x1705107468788672/t0(0) o400->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626195253 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [36690.049887] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [36690.049910] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [36690.049911] Lustre: Skipped 2 previous similar messages [36690.843949] e1000 0000:00:08.0 enp0s8: Detected Tx Unit Hang Tx Queue <0> TDH <0> TDT <4> next_to_use <4> next_to_clean <0> buffer_info[next_to_clean] time_stamp <1022b31a1> next_to_watch <1> jiffies <1022b4951> next_to_watch.status <0> [36692.305282] e1000 0000:00:08.0 enp0s8: Reset adapter [36692.605746] Lustre: 3092:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626195246/real 1626195246] req@ffff9c87572b7500 x1705107468788288/t0(0) o13->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626195253 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [36692.605758] Lustre: 3092:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [36694.356019] e1000: enp0s8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [36696.864068] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [36696.864071] Lustre: Skipped 8 previous similar messages [36696.864137] Lustre: lustrefs-MDT0000: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [36696.864149] Lustre: Skipped 11 previous similar messages [36696.864166] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [36699.714117] 12:37:36.877749 timesync vgsvcTimeSyncWorker: Radical guest time change: 243 808 441 472 000ns (GuestNow=1 626 439 056 877 738 000 ns GuestLast=1 626 195 248 436 266 000 ns fSetTimeLastLoop=true ) [36800.038065] Lustre: 3094:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626439150/real 1626439150] req@ffff9c869e445580 x1705107468810112/t0(0) o400->lustrefs-OST0001-osc-MDT0001@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626439157 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [36800.038091] Lustre: lustrefs-OST0001-osc-MDT0001: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [36800.038092] Lustre: Skipped 3 previous similar messages [36800.039227] Lustre: lustrefs-OST0001-osc-MDT0001: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [36800.039230] Lustre: Skipped 14 previous similar messages [36806.597057] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [36806.597061] Lustre: Skipped 2 previous similar messages [36827.586889] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [36827.586891] Lustre: Skipped 1 previous similar message [36827.586930] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [36827.586933] Lustre: Skipped 7 previous similar messages [37043.015000] Lustre: lustrefs-MDT0001: Client lustrefs-MDT0001-lwp-OST0001_UUID (at 10.0.2.5@tcp) reconnecting [37043.015004] Lustre: Skipped 1 previous similar message [37043.015054] Lustre: lustrefs-MDT0001: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [37043.015055] Lustre: Skipped 6 previous similar messages [37043.060774] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626439393/real 1626439393] req@ffff9c86d49d5580 x1705107468857984/t0(0) o13->lustrefs-OST0001-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626439400 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [37043.060778] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [37043.060809] Lustre: lustrefs-OST0001-osc-MDT0001: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [37382.329885] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626439732/real 1626439732] req@ffff9c86d49e0900 x1705107468925120/t0(0) o400->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626439739 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [37382.329894] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [37382.329920] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [37382.329921] Lustre: Skipped 2 previous similar messages [37382.521451] Lustre: lustrefs-MDT0001: Client lustrefs-MDT0001-lwp-OST0000_UUID (at 10.0.2.5@tcp) reconnecting [37382.521462] Lustre: Skipped 4 previous similar messages [37382.521506] Lustre: lustrefs-MDT0001: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [37382.521507] Lustre: Skipped 7 previous similar messages [37383.760230] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [39062.709045] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626441412/real 1626441412] req@ffff9c87405e6c00 x1705107469248576/t0(0) o13->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626441419 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [39062.709049] Lustre: 3093:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [39062.709070] Lustre: lustrefs-OST0000-osc-MDT0001: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [39062.709071] Lustre: Skipped 2 previous similar messages [39062.710069] Lustre: lustrefs-OST0000-osc-MDT0001: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [39062.710071] Lustre: Skipped 7 previous similar messages [39063.738063] Lustre: lustrefs-MDT0000: Client lustrefs-MDT0000-lwp-OST0000_UUID (at 10.0.2.5@tcp) reconnecting [39063.738065] Lustre: Skipped 3 previous similar messages [39065.882357] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [39066.750146] Lustre: 3092:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626441416/real 1626441416] req@ffff9c86d6db3a80 x1705107469248832/t0(0) o13->lustrefs-OST0001-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626441423 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [39066.750172] Lustre: lustrefs-OST0001-osc-MDT0001: Connection to lustrefs-OST0001 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [44756.507267] 14:51:53.666754 control Session 0 is about to close ... [44756.507834] 14:51:53.667070 control Stopping all guest processes ... [44756.507942] 14:51:53.667577 control Closing all guest files ... [44756.509122] 14:51:53.668027 control vbglR3GuestCtrlDetectPeekGetCancelSupport: Supported (#1) [44757.826603] e1000: enp0s8 NIC Link is Down [44757.826631] e1000 0000:00:08.0 enp0s8: Reset adapter [44757.831872] e1000: enp0s3 NIC Link is Down [44759.842667] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [44760.217570] e1000: enp0s8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [44760.995115] Lustre: lustrefs-MDT0001: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [44760.995118] Lustre: Skipped 7 previous similar messages [44760.995213] Lustre: lustrefs-MDT0001: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [44760.995215] Lustre: Skipped 12 previous similar messages [44762.483543] 07:43:37.978102 timesync vgsvcTimeSyncWorker: Radical host time change: 233 508 348 000 000ns (HostNow=1 626 680 617 978 000 000 ns HostLast=1 626 447 109 630 000 000 ns) [44763.667008] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [44764.957281] Lustre: 3092:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626447116/real 1626447116] req@ffff9c86d4c0f500 x1705107470341504/t0(0) o13->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626447123 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [44764.957284] Lustre: 3092:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [44764.957305] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [44764.957307] Lustre: Skipped 2 previous similar messages [44765.958275] Lustre: 3094:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626447116/real 1626447116] req@ffff9c86d4838480 x1705107470341440/t0(0) o13->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:7/4 lens 224/368 e 0 to 1 dl 1626447123 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [44765.958279] Lustre: 3094:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [44765.958301] Lustre: lustrefs-OST0000-osc-MDT0001: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [44765.958302] Lustre: Skipped 2 previous similar messages [44765.962940] Lustre: lustrefs-OST0000-osc-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [44765.962943] Lustre: Skipped 4 previous similar messages [44768.057443] Lustre: MGS: Client 35f03572-7832-4520-ab76-24e2505f6d2b (at 10.0.2.5@tcp) reconnecting [44768.057446] Lustre: Skipped 3 previous similar messages [44770.972752] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626680619/real 1626680619] req@ffff9c86d483d580 x1705107470341952/t0(0) o400->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626680626 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [44770.972756] Lustre: 3095:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [44772.487869] 07:43:47.982472 timesync vgsvcTimeSyncWorker: Radical guest time change: 233 508 339 401 000ns (GuestNow=1 626 680 627 982 461 000 ns GuestLast=1 626 447 119 643 060 000 ns fSetTimeLastLoop=true ) [44824.218848] LNetError: 3084:0:(socklnd.c:1665:ksocknal_destroy_conn()) Completing partial receive from 12345-10.0.2.6@tcp[1], ip 10.0.2.6:1020, with error, wanted: 224, left: 224, last alive is 0 secs ago [44824.219533] LustreError: 3084:0:(events.c:310:request_in_callback()) event type 2, status -5, service mdt [44828.310278] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [44828.310359] Lustre: MGS: Connection restored to 09b330b1-f4d3-af79-ddf2-9a17ac59ec56 (at 10.0.2.6@tcp) [44828.310361] Lustre: Skipped 8 previous similar messages [44828.310369] Lustre: lustrefs-MDT0000: Client ec8e9e21-1054-806c-9707-2a39fc20c80a (at 10.0.2.6@tcp) reconnecting [44828.310371] Lustre: Skipped 4 previous similar messages [44870.959825] Lustre: MGS: Client 35f03572-7832-4520-ab76-24e2505f6d2b (at 10.0.2.5@tcp) reconnecting [44870.959828] Lustre: Skipped 1 previous similar message [44870.959907] Lustre: MGS: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [44870.959909] Lustre: Skipped 2 previous similar messages [44873.247162] Lustre: 3092:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626680721/real 1626680721] req@ffff9c86d1235a00 x1705107470363840/t0(0) o400->lustrefs-OST0000-osc-MDT0001@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626680728 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [44873.247165] Lustre: 3092:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [44873.247189] Lustre: lustrefs-OST0000-osc-MDT0001: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [44874.373004] Lustre: MGS: Received new LWP connection from 10.0.2.6@tcp, removing former export from same NID [44885.305848] Lustre: 3094:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626680733/real 1626680733] req@ffff9c869e53ec00 x1705107470367232/t0(0) o400->lustrefs-OST0000-osc-MDT0000@10.0.2.5@tcp:28/4 lens 224/224 e 0 to 1 dl 1626680740 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [44885.305852] Lustre: 3094:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [44885.305877] Lustre: lustrefs-OST0000-osc-MDT0000: Connection to lustrefs-OST0000 (at 10.0.2.5@tcp) was lost; in progress operations using this service will wait for recovery to complete [44885.305878] Lustre: Skipped 1 previous similar message [44899.584011] LustreError: 3094:0:(client.c:1210:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9c86d1237080 x1705107470372928/t0(0) o41->lustrefs-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [44899.584173] LustreError: 11-0: lustrefs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [44899.584178] Lustre: lustrefs-MDT0000-osp-MDT0001: Connection to lustrefs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [44899.584179] Lustre: Skipped 1 previous similar message [44899.584343] Lustre: lustrefs-MDT0000: Not available for connect from 0@lo (stopping) [44904.633142] Lustre: lustrefs-MDT0000: Not available for connect from 0@lo (stopping) [44904.633149] Lustre: Skipped 1 previous similar message [44904.992279] Lustre: server umount lustrefs-MDT0000 complete [44912.957363] Lustre: server umount lustrefs-MDT0001 complete [44921.001823] Lustre: MGS: Client 35f03572-7832-4520-ab76-24e2505f6d2b (at 10.0.2.5@tcp) reconnecting [44921.001825] Lustre: Skipped 5 previous similar messages [44921.001858] Lustre: MGS: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [44921.001859] Lustre: Skipped 10 previous similar messages [45112.685265] Lustre: 23041:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626680962/real 1626680962] req@ffff9c869e53d100 x1705107470377216/t0(0) o251->MGC10.0.2.4@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1626680968 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [45112.685270] Lustre: 23041:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 1 previous similar message [45112.731969] Lustre: server umount MGS complete [45757.496893] LNet: Removed LNI 10.0.2.104@tcp1 [45759.507177] LNet: Removed LNI 10.0.2.4@tcp [45854.655758] LNet: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [45854.658784] alg: No test for adler32 (adler32-zlib) [45855.424910] Lustre: DEBUG MARKER: mds1: executing set_hostid [45869.741482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [45874.074972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [45877.575540] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46004.949023] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46005.088292] Lustre: Lustre: Build Version: 2.12.7_RC1 [46005.179737] LNet: Added LNI 10.0.2.4@tcp [8/256/0/180] [46005.179743] LNet: Added LNI 10.0.2.104@tcp1 [8/256/0/180] [46005.179787] LNet: Accept secure, port 988 [46005.439135] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [46005.470149] Lustre: MGS: Connection restored to MGC10.0.2.4@tcp_0 (at 0@lo) [46006.001534] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [46008.650941] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46008.729944] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [46008.794927] Lustre: Setting parameter lustrefs-MDT0000.mdt.identity_upcall in log lustrefs-MDT0000 [46008.825112] Lustre: ctl-lustrefs-MDT0000: No data found on store. Initialize space [46008.856542] Lustre: lustrefs-MDT0000: new disk, initializing [46008.958690] Lustre: lustrefs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [46008.969994] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [46009.728701] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [46013.919698] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46013.945604] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [46014.004129] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [46014.027490] Lustre: Setting parameter lustrefs-MDT0001.mdt.identity_upcall in log lustrefs-MDT0001 [46014.039003] Lustre: srv-lustrefs-MDT0001: No data found on store. Initialize space [46014.039006] Lustre: Skipped 1 previous similar message [46014.072636] Lustre: lustrefs-MDT0001: new disk, initializing [46014.109416] Lustre: lustrefs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [46014.121032] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [46014.121147] Lustre: cli-ctl-lustrefs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [46014.756240] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [46019.983475] Lustre: lustrefs-MDT0001: Connection restored to 10.0.2.4@tcp (at 0@lo) [46019.983478] Lustre: Skipped 2 previous similar messages [46300.376993] LustreError: 11-0: lustrefs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [46300.377716] Lustre: lustrefs-MDT0000-osp-MDT0001: Connection to lustrefs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [46300.377889] Lustre: lustrefs-MDT0000: Not available for connect from 0@lo (stopping) [46300.824668] LustreError: 24821:0:(client.c:1210:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9c86b1520000 x1705699558063424/t0(0) o41->lustrefs-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [46304.921213] Lustre: server umount lustrefs-MDT0000 complete [46305.384042] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [46305.384877] LustreError: Skipped 1 previous similar message [46306.662155] Lustre: server umount lustrefs-MDT0001 complete [46437.181330] Lustre: 26751:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626682286/real 1626682286] req@ffff9c86b1520480 x1705699558065600/t0(0) o251->MGC10.0.2.4@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1626682292 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [46437.213900] Lustre: server umount MGS complete [46547.288929] LNet: Removed LNI 10.0.2.104@tcp1 [46548.315761] LNet: Removed LNI 10.0.2.4@tcp [46644.251910] LNet: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [46644.256534] alg: No test for adler32 (adler32-zlib) [46645.017424] Lustre: DEBUG MARKER: mds1: executing set_hostid [46659.307923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46664.040470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46668.604870] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46794.587471] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46794.858888] Lustre: Lustre: Build Version: 2.12.7_RC1 [46794.930476] LNet: Added LNI 10.0.2.4@tcp [8/256/0/180] [46794.930481] LNet: Added LNI 10.0.2.104@tcp1 [8/256/0/180] [46794.930531] LNet: Accept secure, port 988 [46796.660245] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [46796.690344] Lustre: MGS: Connection restored to f242e177-351a-713b-3ade-d7345711203b (at 0@lo) [46797.241511] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [46800.334668] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46800.414259] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [46800.467531] Lustre: Setting parameter lustrefs-MDT0000.mdt.identity_upcall in log lustrefs-MDT0000 [46800.495307] Lustre: ctl-lustrefs-MDT0000: No data found on store. Initialize space [46800.527425] Lustre: lustrefs-MDT0000: new disk, initializing [46800.666425] Lustre: lustrefs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [46800.677965] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [46801.302637] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [46804.956197] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: errors=remount-ro [46805.094640] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [46805.118974] Lustre: Setting parameter lustrefs-MDT0001.mdt.identity_upcall in log lustrefs-MDT0001 [46805.132199] Lustre: srv-lustrefs-MDT0001: No data found on store. Initialize space [46805.132208] Lustre: Skipped 1 previous similar message [46805.161261] Lustre: lustrefs-MDT0001: new disk, initializing [46805.173477] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [46805.203343] Lustre: lustrefs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [46805.214889] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [46805.214920] Lustre: cli-ctl-lustrefs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [46805.847723] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [46813.390128] Lustre: lustrefs-MDT0001: Connection restored to 10.0.2.4@tcp (at 0@lo) [46813.390132] Lustre: Skipped 2 previous similar messages [46911.097237] Lustre: MGS: Connection restored to 9f300d88-9a38-e052-59b8-aef2a3778088 (at 10.0.2.5@tcp) [46919.209356] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [47077.308095] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [47077.308107] Lustre: Skipped 2 previous similar messages [47086.485254] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [47140.198522] Lustre: MGS: Connection restored to a084ee9f-1c93-8331-e423-1d3dfd74f167 (at 10.0.2.6@tcp) [47140.198525] Lustre: Skipped 1 previous similar message [47163.779256] Lustre: DEBUG MARKER: Using TIMEOUT=20 [60938.777566] Lustre: Failing over lustrefs-MDT0000 [60938.940172] Lustre: server umount lustrefs-MDT0000 complete [60939.698927] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [60939.698928] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [60940.751438] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [60942.271359] LustreError: 11-0: lustrefs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [60942.271704] Lustre: lustrefs-MDT0000-osp-MDT0001: Connection to lustrefs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [60942.272676] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [60944.714046] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [60944.715240] LustreError: Skipped 2 previous similar messages [60949.722143] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [60949.723535] LustreError: Skipped 4 previous similar messages [60959.738228] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [60959.739637] LustreError: Skipped 9 previous similar messages [60975.814062] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [60975.815443] LustreError: Skipped 15 previous similar messages [60998.866096] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null) [61008.188979] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [61008.190454] LustreError: Skipped 31 previous similar messages [61073.341398] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [61073.342793] LustreError: Skipped 63 previous similar messages [61202.036166] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [61202.037504] LustreError: Skipped 127 previous similar messages [61460.569895] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.2.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [61460.571234] LustreError: Skipped 255 previous similar messages [61794.328911] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [61794.429590] Lustre: lustrefs-MDT0001: Connection restored to 10.0.2.4@tcp (at 0@lo) [61794.429593] Lustre: Skipped 2 previous similar messages [61794.476336] Lustre: lustrefs-MDT0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [61794.488913] Lustre: lustrefs-MDT0000: in recovery but waiting for the first client to connect [61795.341020] Lustre: lustrefs-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [61799.474623] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [61799.474626] Lustre: Skipped 3 previous similar messages [61799.475100] LustreError: 167-0: lustrefs-MDT0000-lwp-MDT0001: This client was evicted by lustrefs-MDT0000; in progress operations using this service will fail. [61799.491455] Lustre: lustrefs-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [62005.206427] LustreError: 28343:0:(client.c:1210:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9c86db1b9f80 x1705700389239104/t0(0) o41->lustrefs-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [62005.206613] Lustre: lustrefs-MDT0000-lwp-MDT0001: Connection to lustrefs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [62005.206614] Lustre: Skipped 1 previous similar message [62005.206709] Lustre: lustrefs-MDT0000: Not available for connect from 0@lo (stopping) [62005.208529] LustreError: 11-0: lustrefs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [62007.028397] Lustre: lustrefs-MDT0000: Not available for connect from 10.0.2.5@tcp (stopping) [62007.028430] Lustre: Skipped 1 previous similar message [62008.636228] Lustre: server umount lustrefs-MDT0000 complete [62010.236864] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [62010.238366] LustreError: Skipped 331 previous similar messages [62022.058858] Lustre: lustrefs-MDT0001: haven't heard from client 7de4c5cd-f330-a541-6d94-6a86e4cdb2a9 (at 10.0.2.6@tcp) in 51 seconds. I think it's dead, and I am evicting it. exp ffff9c874cac6800, cur 1626697877 expire 1626697847 last 1626697826 [62100.421415] Lustre: server umount lustrefs-MDT0001 complete [62384.086072] Lustre: 15164:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626698233/real 1626698233] req@ffff9c8697251f80 x1705700389252224/t0(0) o251->MGC10.0.2.4@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1626698239 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [62384.128279] Lustre: server umount MGS complete [62517.270547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [62517.315175] Lustre: MGS: Connection restored to MGC10.0.2.4@tcp_0 (at 0@lo) [62517.315177] Lustre: Skipped 3 previous similar messages [62537.133355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [62537.286565] Lustre: lustrefs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [62542.260991] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [62548.959916] LustreError: 15483:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on /dev/mapper/VGLustre-mdt1: -114 [62548.961204] LustreError: 15483:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-114) [62555.664596] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [62555.753152] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [62555.778668] Lustre: lustrefs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [62560.765643] Lustre: lustrefs-MDT0001: Connection restored to 10.0.2.4@tcp (at 0@lo) [62560.765646] Lustre: Skipped 1 previous similar message [62971.153460] Lustre: Failing over lustrefs-MDT0001 [62971.272251] Lustre: server umount lustrefs-MDT0001 complete [62971.482034] LustreError: 11-0: lustrefs-MDT0001-osp-MDT0000: operation mds_disconnect to node 0@lo failed: rc = -107 [62977.883630] Lustre: 17847:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626698827/real 1626698827] req@ffff9c86d1e5bf00 x1705700389307520/t0(0) o251->MGC10.0.2.4@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1626698833 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [62977.922502] Lustre: server umount MGS complete [62977.922505] Lustre: Skipped 1 previous similar message [63190.712188] Lustre: DEBUG MARKER: mds1: executing set_hostid [63244.524653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [63298.645805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [63352.595917] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [63608.885278] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [63608.958854] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [63608.982063] Lustre: MGS: Connection restored to 10.0.2.4@tcp (at 0@lo) [63629.580032] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [63762.380377] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro [63762.644654] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [63762.679100] Lustre: Setting parameter lustrefs-MDT0000.mdt.identity_upcall in log lustrefs-MDT0000 [63762.686718] Lustre: ctl-lustrefs-MDT0000: No data found on store. Initialize space [63762.714610] Lustre: lustrefs-MDT0000: new disk, initializing [63762.789055] Lustre: lustrefs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [63762.796432] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [63767.839641] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [63783.448646] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [63927.484273] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: errors=remount-ro [63927.658585] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [63927.680158] Lustre: Setting parameter lustrefs-MDT0001.mdt.identity_upcall in log lustrefs-MDT0001 [63927.691831] Lustre: srv-lustrefs-MDT0001: No data found on store. Initialize space [63927.691833] Lustre: Skipped 1 previous similar message [63927.718956] Lustre: lustrefs-MDT0001: new disk, initializing [63927.732751] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [63927.761389] Lustre: lustrefs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [63927.772920] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [63927.772957] Lustre: cli-ctl-lustrefs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [63933.105488] Lustre: lustrefs-MDT0001: Connection restored to 10.0.2.4@tcp (at 0@lo) [63933.105491] Lustre: Skipped 1 previous similar message [63948.371482] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [64093.076353] Lustre: MGS: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [64100.634513] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [64183.855794] LustreError: 28343:0:(client.c:1210:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9c86a98b7080 x1705700389350720/t0(0) o41->lustrefs-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [64185.839625] Lustre: lustrefs-MDT0000-osp-MDT0001: Connection to lustrefs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [64185.839629] Lustre: Skipped 1 previous similar message [64185.839792] Lustre: lustrefs-MDT0000: Not available for connect from 0@lo (stopping) [64185.839794] Lustre: Skipped 1 previous similar message [64188.677701] Lustre: lustrefs-MDT0000: Not available for connect from 10.0.2.5@tcp (stopping) [64188.677720] Lustre: Skipped 1 previous similar message [64189.835533] Lustre: server umount lustrefs-MDT0000 complete [64190.846731] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [64190.848173] LustreError: Skipped 76 previous similar messages [64192.189181] Lustre: server umount lustrefs-MDT0001 complete [64209.877130] Lustre: 21140:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626700059/real 1626700059] req@ffff9c874017a400 x1705700389352256/t0(0) o251->MGC10.0.2.4@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1626700065 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [64209.911136] Lustre: server umount MGS complete [64247.968253] Lustre: DEBUG MARKER: mds1: executing set_hostid [64251.850397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [64256.550647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [64260.435734] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [64267.633642] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [64267.703914] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [64267.727054] Lustre: MGS: Connection restored to MGC10.0.2.4@tcp_0 (at 0@lo) [64267.727057] Lustre: Skipped 2 previous similar messages [64268.466964] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [64271.102865] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro [64271.262603] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [64271.296683] Lustre: Setting parameter lustrefs-MDT0000.mdt.identity_upcall in log lustrefs-MDT0000 [64271.310685] Lustre: ctl-lustrefs-MDT0000: No data found on store. Initialize space [64271.339569] Lustre: lustrefs-MDT0000: new disk, initializing [64271.438241] Lustre: lustrefs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [64271.450601] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [64272.036911] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [64275.713792] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: errors=remount-ro [64275.848762] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [64275.870781] Lustre: Setting parameter lustrefs-MDT0001.mdt.identity_upcall in log lustrefs-MDT0001 [64275.880246] Lustre: srv-lustrefs-MDT0001: No data found on store. Initialize space [64275.880249] Lustre: Skipped 1 previous similar message [64275.908101] Lustre: lustrefs-MDT0001: new disk, initializing [64276.320586] Lustre: lustrefs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [64276.333134] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [64276.333185] Lustre: cli-ctl-lustrefs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [64277.057634] Lustre: DEBUG MARKER: mds1: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [64286.152703] Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [64288.032070] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.5@tcp (at 10.0.2.5@tcp) [64288.032076] Lustre: Skipped 7 previous similar messages [64293.963476] Lustre: DEBUG MARKER: Using TIMEOUT=20 [64330.678050] Lustre: Failing over lustrefs-MDT0001 [64330.678053] Lustre: Skipped 1 previous similar message [64330.816337] Lustre: server umount lustrefs-MDT0001 complete [64330.884301] LustreError: 11-0: lustrefs-MDT0001-osp-MDT0000: operation mds_disconnect to node 0@lo failed: rc = -107 [64337.103949] Lustre: 24103:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1626700186/real 1626700186] req@ffff9c871b3df980 x1705700389382272/t0(0) o251->MGC10.0.2.4@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1626700192 ref 2 fl Rpc:IXN/0/ffffffff rc 0/-1 [64384.063516] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [64384.103076] Lustre: MGS: Connection restored to MGC10.0.2.4@tcp_0 (at 0@lo) [64384.103078] Lustre: Skipped 4 previous similar messages [64418.592237] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null) [64544.778789] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [64544.882943] LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [64544.958033] Lustre: lustrefs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [64544.960635] Lustre: lustrefs-MDT0001: in recovery but waiting for the first client to connect [64551.806749] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [64551.905929] LustreError: 11-0: lustrefs-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [64552.003176] Lustre: lustrefs-MDT0000: in recovery but waiting for the first client to connect [64556.993276] Lustre: lustrefs-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [64556.993338] Lustre: lustrefs-MDT0000: Connection restored to 10.0.2.4@tcp (at 0@lo) [64556.993341] Lustre: Skipped 2 previous similar messages [64569.324795] Lustre: lustrefs-MDT0001: Recovery over after 0:12, of 2 clients 2 recovered and 0 were evicted. [65226.320307] LustreError: 24497:0:(osd_handler.c:1368:osd_fid_lookup()) ASSERTION( !updated ) failed: [65226.321549] LustreError: 24497:0:(osd_handler.c:1368:osd_fid_lookup()) LBUG [65226.322160] Pid: 24497, comm: mdt00_004 3.10.0-1127.8.2.el7_lustre.x86_64 #1 SMP Mon Jun 8 13:48:45 UTC 2020 [65226.322161] Call Trace: [65226.322171] [<ffffffffc09c57cc>] libcfs_call_trace+0x8c/0xc0 [libcfs] [65226.322180] [<ffffffffc09c587c>] lbug_with_loc+0x4c/0xa0 [libcfs] [65226.322184] [<ffffffffc113c4e1>] osd_fid_lookup+0x17e1/0x1d90 [osd_ldiskfs] [65226.322192] [<ffffffffc113caf1>] osd_object_init+0x61/0x110 [osd_ldiskfs] [65226.322198] [<ffffffffc0c0d82b>] lu_object_start.isra.35+0x8b/0x120 [obdclass] [65226.322221] [<ffffffffc0c11744>] lu_object_find_at+0x234/0xab0 [obdclass] [65226.322235] [<ffffffffc0c11fd6>] lu_object_find+0x16/0x20 [obdclass] [65226.322248] [<ffffffffc12e146b>] mdt_object_find+0x4b/0x170 [mdt] [65226.322259] [<ffffffffc12ed0a5>] mdt_rpc_fid2path+0x4a5/0x17c0 [mdt] [65226.322267] [<ffffffffc12ee770>] mdt_get_info+0x3b0/0x490 [mdt] [65226.322275] [<ffffffffc0f5489a>] tgt_request_handle+0xada/0x1570 [ptlrpc] [65226.322319] [<ffffffffc0ef973b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [65226.322343] [<ffffffffc0efd0a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc] [65226.322366] [<ffffffffa2ac6691>] kthread+0xd1/0xe0 [65226.322369] [<ffffffffa3192d37>] ret_from_fork_nospec_end+0x0/0x39 [65226.322372] [<ffffffffffffffff>] 0xffffffffffffffff [65226.322385] Kernel panic - not syncing: LBUG [65226.323010] CPU: 0 PID: 24497 Comm: mdt00_004 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1127.8.2.el7_lustre.x86_64 #1 [65226.324423] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [65226.325029] Call Trace: [65226.325686] [<ffffffffa317ffa5>] dump_stack+0x19/0x1b [65226.326279] [<ffffffffa3179541>] panic+0xe8/0x21f [65226.326863] [<ffffffffc09c58cb>] lbug_with_loc+0x9b/0xa0 [libcfs] [65226.327452] [<ffffffffc113c4e1>] osd_fid_lookup+0x17e1/0x1d90 [osd_ldiskfs] [65226.328022] [<ffffffffc09cc177>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [65226.328590] [<ffffffffc113caf1>] osd_object_init+0x61/0x110 [osd_ldiskfs] [65226.329180] [<ffffffffc0c0d82b>] lu_object_start.isra.35+0x8b/0x120 [obdclass] [65226.329742] [<ffffffffc0c11744>] lu_object_find_at+0x234/0xab0 [obdclass] [65226.330294] [<ffffffffc09cc177>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [65226.330834] [<ffffffffc0c11fd6>] lu_object_find+0x16/0x20 [obdclass] [65226.331371] [<ffffffffc12e146b>] mdt_object_find+0x4b/0x170 [mdt] [65226.331936] [<ffffffffc12ed0a5>] mdt_rpc_fid2path+0x4a5/0x17c0 [mdt] [65226.332489] [<ffffffffc09c8a94>] ? libcfs_log_return+0x24/0x30 [libcfs] [65226.333034] [<ffffffffc0eec829>] ? lustre_pack_reply_v2+0x169/0x290 [ptlrpc] [65226.333592] [<ffffffffc0eea090>] ? lustre_msg_buf_v2+0x1e0/0x1e0 [ptlrpc] [65226.334152] [<ffffffffc0eea0a7>] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [65226.334795] [<ffffffffc0f11323>] ? __req_capsule_get+0x163/0x740 [ptlrpc] [65226.335329] [<ffffffffc0eec9bf>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [65226.335804] [<ffffffffc0eeab32>] ? lustre_msg_get_transno+0x22/0xf0 [ptlrpc] [65226.336310] [<ffffffffc12e7354>] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [65226.336754] [<ffffffffc12ee770>] mdt_get_info+0x3b0/0x490 [mdt] [65226.337214] [<ffffffffc0f5489a>] tgt_request_handle+0xada/0x1570 [ptlrpc] [65226.337659] [<ffffffffc09cc177>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [65226.338108] [<ffffffffc0ef973b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [65226.338594] [<ffffffffc09c8a94>] ? libcfs_log_return+0x24/0x30 [libcfs] [65226.339055] [<ffffffffc0efd0a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc] [65226.339482] [<ffffffffa2ad53fe>] ? finish_task_switch+0x4e/0x1c0 [65226.339889] [<ffffffffa3185942>] ? __schedule+0x402/0x840 [65226.340351] [<ffffffffc0efc570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [65226.340765] [<ffffffffa2ac6691>] kthread+0xd1/0xe0 [65226.341152] [<ffffffffa2ac65c0>] ? insert_kthread_work+0x40/0x40 [65226.341541] [<ffffffffa3192d37>] ret_from_fork_nospec_begin+0x21/0x21 [65226.341921] [<ffffffffa2ac65c0>] ? insert_kthread_work+0x40/0x40