[ 0.000000] Linux version 4.18.0-348.2.1.el8_lustre.x86_64 (jenkins@onyx-199-el8-x8664-2.onyx.whamcloud.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-4) (GCC)) #1 SMP Sun Apr 3 16:16:31 UTC 2022 [ 0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-4.18.0-348.2.1.el8_lustre.x86_64 root=UUID=81a077d7-2ba3-4dc6-b915-ceb8a4f28f88 ro crashkernel=auto resume=UUID=ad7420b6-f6e3-46d3-948d-61da8eb250ec console=ttyS0,115200 console=tty0 console=ttyS0,115200 crashkernel=320M net.ifnames=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bfff9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bfffa000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] DMI: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 82a01001, primary cpu clock [ 0.000000] kvm-clock: using sched offset of 25401055594 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2693.508 MHz processor [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] last_pfn = 0xbfffa max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] found SMP MP-table at [mem 0x000f6390-0x000f639f] [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x82c01000, 0x82c01fff] PGTABLE [ 0.000000] BRK [0x82c02000, 0x82c02fff] PGTABLE [ 0.000000] BRK [0x82c03000, 0x82c03fff] PGTABLE [ 0.000000] BRK [0x82c04000, 0x82c04fff] PGTABLE [ 0.000000] BRK [0x82c05000, 0x82c05fff] PGTABLE [ 0.000000] BRK [0x82c06000, 0x82c06fff] PGTABLE [ 0.000000] RAMDISK: [mem 0x34c97000-0x36643fff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F61C0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFFFB40 000030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFFF177 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFFE040 001137 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFFE000 000040 [ 0.000000] ACPI: SSDT 0x00000000BFFFF1EB 0008D5 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 0x00000000BFFFFAC0 000080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffff177-0xbffff1ea] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbfffe040-0xbffff176] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbfffe000-0xbfffe03f] [ 0.000000] ACPI: Reserving SSDT table memory at [mem 0xbffff1eb-0xbffffabf] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffffac0-0xbffffb3f] [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x00000000bfff9fff] [ 0.000000] NODE_DATA(0) allocated [mem 0xbffcf000-0xbfff9fff] [ 0.000000] Reserving 320MB of memory at 2736MB for crashkernel (System RAM: 3071MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000bfff9fff] [ 0.000000] Normal empty [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bfff9fff] [ 0.000000] Zeroed struct page in unavailable ranges: 104 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000000bfff9fff] [ 0.000000] On node 0 totalpages: 786328 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 158 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782330 pages, LIFO batch:63 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] percpu: Embedded 54 pages/cpu s184320 r8192 d28672 u1048576 [ 0.000000] pcpu-alloc: s184320 r8192 d28672 u1048576 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 [ 0.000000] kvm-guest: stealtime: cpu 0, msr bfc2c080 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 773882 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-4.18.0-348.2.1.el8_lustre.x86_64 root=UUID=81a077d7-2ba3-4dc6-b915-ceb8a4f28f88 ro crashkernel=auto resume=UUID=ad7420b6-f6e3-46d3-948d-61da8eb250ec console=ttyS0,115200 console=tty0 console=ttyS0,115200 crashkernel=320M net.ifnames=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] Memory: 261120K/3145312K available (12293K kernel code, 2261K rwdata, 7872K rodata, 2492K init, 13944K bss, 449544K reserved, 0K cma-reserved) [ 0.000000] random: get_random_u64 called from cache_random_seq_create+0x7c/0x140 with crng_init=0 [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Kernel/User page tables isolation: enabled [ 0.000000] ftrace: allocating 39482 entries in 155 pages [ 0.000000] ftrace: allocated 155 pages with 5 groups [ 0.001000] rcu: Hierarchical RCU implementation. [ 0.001000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=2. [ 0.001000] Rude variant of Tasks RCU enabled. [ 0.001000] Tracing variant of Tasks RCU enabled. [ 0.001000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.001000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 [ 0.001000] NR_IRQS: 524544, nr_irqs: 440, preallocated irqs: 16 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [tty0] enabled [ 0.001000] printk: console [ttyS0] enabled [ 0.001000] ACPI: Core revision 20210105 [ 0.001004] APIC: Switch to symmetric I/O mode setup [ 0.002311] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.005787] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x26d349e8249, max_idle_ns: 440795288087 ns [ 0.007003] Calibrating delay loop (skipped) preset value.. 5387.01 BogoMIPS (lpj=2693508) [ 0.008002] pid_max: default: 32768 minimum: 301 [ 0.009041] LSM: Security Framework initializing [ 0.010020] Yama: becoming mindful. [ 0.011009] SELinux: Initializing. [ 0.012062] LSM support for eBPF active [ 0.018555] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022842] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, vmalloc) [ 0.024094] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, vmalloc) [ 0.026091] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, vmalloc) [ 0.028403] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030003] FEATURE SPEC_CTRL Present [ 0.031001] FEATURE IBPB_SUPPORT Present [ 0.032002] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.034003] Spectre V2 : Mitigation: Full generic retpoline [ 0.035002] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.036001] Spectre V2 : Enabling Restricted Speculation for firmware calls [ 0.038003] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier [ 0.040002] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.041005] MDS: Mitigation: Clear CPU buffers [ 0.043115] Freeing SMP alternatives memory: 32K [ 0.046357] TSC deadline timer enabled [ 0.046361] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge, IBRS update) (family: 0x6, model: 0x2a, stepping: 0x1) [ 0.047318] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.048111] rcu: Hierarchical SRCU implementation. [ 0.049848] NMI watchdog: Perf NMI watchdog permanently disabled [ 0.050181] smp: Bringing up secondary CPUs ... [ 0.051393] x86: Booting SMP configuration: [ 0.052007] .... node #0, CPUs: #1 [ 0.001000] kvm-clock: cpu 1, msr 82a01041, secondary cpu clock [ 0.001000] smpboot: CPU 1 Converting physical 0 to logical die 1 [ 0.056025] kvm-guest: stealtime: cpu 1, msr bfd2c080 [ 0.057010] smp: Brought up 1 node, 2 CPUs [ 0.057803] smpboot: Max logical packages: 2 [ 0.058008] smpboot: Total of 2 processors activated (10774.03 BogoMIPS) [ 0.090820] node 0 deferred pages initialised in 31ms [ 0.092281] devtmpfs: initialized [ 0.093095] x86/mm: Memory block size: 128MB [ 0.094487] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.095028] futex hash table entries: 512 (order: 3, 32768 bytes, vmalloc) [ 0.096114] pinctrl core: initialized pinctrl subsystem [ 0.097387] NET: Registered protocol family 16 [ 0.098168] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.099017] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.100018] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.101045] audit: initializing netlink subsys (disabled) [ 0.102069] audit: type=2000 audit(1651508641.541:1): state=initialized audit_enabled=0 res=1 [ 0.103025] cpuidle: using governor menu [ 0.104156] ACPI: bus type PCI registered [ 0.104937] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.105237] PCI: Using configuration type 1 for base access [ 0.108108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.109007] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.110109] cryptd: max_cpu_qlen set to 1000 [ 0.111079] ACPI: Added _OSI(Module Device) [ 0.112012] ACPI: Added _OSI(Processor Device) [ 0.112864] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.113005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.114005] ACPI: Added _OSI(Linux-Dell-Video) [ 0.114854] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio) [ 0.115005] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics) [ 0.116869] ACPI: 2 ACPI AML tables successfully acquired and loaded [ 0.117834] ACPI: Interpreter enabled [ 0.118021] ACPI: (supports S0 S5) [ 0.118695] ACPI: Using IOAPIC for interrupt routing [ 0.119026] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.120298] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.123892] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.124011] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI EDR HPX-Type3] [ 0.125016] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.126407] acpiphp: Slot [3] registered [ 0.127034] acpiphp: Slot [4] registered [ 0.127830] acpiphp: Slot [5] registered [ 0.128043] acpiphp: Slot [6] registered [ 0.128835] acpiphp: Slot [7] registered [ 0.129033] acpiphp: Slot [8] registered [ 0.129824] acpiphp: Slot [9] registered [ 0.130033] acpiphp: Slot [10] registered [ 0.130838] acpiphp: Slot [11] registered [ 0.131041] acpiphp: Slot [12] registered [ 0.131845] acpiphp: Slot [13] registered [ 0.132033] acpiphp: Slot [14] registered [ 0.132842] acpiphp: Slot [15] registered [ 0.133033] acpiphp: Slot [16] registered [ 0.133841] acpiphp: Slot [17] registered [ 0.134045] acpiphp: Slot [18] registered [ 0.134849] acpiphp: Slot [19] registered [ 0.135033] acpiphp: Slot [20] registered [ 0.135835] acpiphp: Slot [21] registered [ 0.136032] acpiphp: Slot [22] registered [ 0.136832] acpiphp: Slot [23] registered [ 0.137041] acpiphp: Slot [24] registered [ 0.137849] acpiphp: Slot [25] registered [ 0.138032] acpiphp: Slot [26] registered [ 0.138837] acpiphp: Slot [27] registered [ 0.139032] acpiphp: Slot [28] registered [ 0.139832] acpiphp: Slot [29] registered [ 0.140042] acpiphp: Slot [30] registered [ 0.140847] acpiphp: Slot [31] registered [ 0.141023] PCI host bridge to bus 0000:00 [ 0.141814] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.142006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.143006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.144006] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.145006] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.146074] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.147653] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.148877] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.154006] pci 0000:00:01.1: reg 0x20: [io 0xc0a0-0xc0af] [ 0.157043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.158006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.159005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.160005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.161294] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300 [ 0.167006] pci 0000:00:01.2: reg 0x20: [io 0xc040-0xc05f] [ 0.170279] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.171645] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.172019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.173429] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000 [ 0.176006] pci 0000:00:03.0: reg 0x10: [io 0xc060-0xc07f] [ 0.179006] pci 0000:00:03.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.190006] pci 0000:00:03.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.191637] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 [ 0.194006] pci 0000:00:04.0: reg 0x10: [io 0xc000-0xc03f] [ 0.197009] pci 0000:00:04.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.208627] pci 0000:00:05.0: [1af4:1002] type 00 class 0x00ff00 [ 0.210006] pci 0000:00:05.0: reg 0x10: [io 0xc080-0xc09f] [ 0.217889] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.218134] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.219120] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.220118] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.221069] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.222294] iommu: Default domain type: Passthrough [ 0.223057] vgaarb: loaded [ 0.223734] SCSI subsystem initialized [ 0.224062] ACPI: bus type USB registered [ 0.224862] usbcore: registered new interface driver usbfs [ 0.225033] usbcore: registered new interface driver hub [ 0.226026] usbcore: registered new device driver usb [ 0.227015] pps_core: LinuxPPS API ver. 1 registered [ 0.227946] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.228010] PTP clock support registered [ 0.228844] EDAC MC: Ver: 3.0.0 [ 0.229326] PCI: Using ACPI for IRQ routing [ 0.230007] PCI: pci_cache_line_size set to 64 bytes [ 0.230197] e820: reserve RAM buffer [mem 0x0009f800-0x0009ffff] [ 0.230199] e820: reserve RAM buffer [mem 0xbfffa000-0xbfffffff] [ 0.230391] NetLabel: Initializing [ 0.231005] NetLabel: domain hash size = 128 [ 0.231840] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.232023] NetLabel: unlabeled traffic allowed by default [ 0.233101] clocksource: Switched to clocksource kvm-clock [ 0.259089] VFS: Disk quotas dquot_6.6.0 [ 0.259923] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.261263] pnp: PnP ACPI init [ 0.261936] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.261971] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.262000] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.262039] pnp 00:03: [dma 2] [ 0.262056] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.262140] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.262346] pnp: PnP ACPI: found 5 devices [ 0.270950] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.272630] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.273792] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.274951] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.276243] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.277576] NET: Registered protocol family 2 [ 0.281467] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, vmalloc) [ 0.283129] TCP established hash table entries: 32768 (order: 6, 262144 bytes, vmalloc) [ 0.284828] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, vmalloc) [ 0.286761] TCP: Hash tables configured (established 32768 bind 32768) [ 0.288324] MPTCP token hash table entries: 4096 (order: 4, 98304 bytes, vmalloc) [ 0.289740] UDP hash table entries: 2048 (order: 4, 65536 bytes, vmalloc) [ 0.291095] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, vmalloc) [ 0.292588] NET: Registered protocol family 1 [ 0.293447] NET: Registered protocol family 44 [ 0.294312] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.295460] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.296560] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.298104] PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.299902] PCI: CLS 0 bytes, default 64 [ 0.300777] Unpacking initramfs... [ 0.768250] Freeing initrd memory: 26292K [ 0.769316] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x26d349e8249, max_idle_ns: 440795288087 ns [ 0.790822] Initialise system trusted keyrings [ 0.791719] Key type blacklist registered [ 0.792564] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 0.796986] zbud: loaded [ 0.797950] pstore: using deflate compression [ 0.799273] Platform Keyring initialized [ 0.929923] NET: Registered protocol family 38 [ 0.930819] Key type asymmetric registered [ 0.931632] Asymmetric key parser 'x509' registered [ 0.932578] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 0.934042] io scheduler mq-deadline registered [ 0.934909] io scheduler kyber registered [ 0.935746] io scheduler bfq registered [ 0.936593] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 0.938068] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 0.939447] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.940863] ACPI: Power Button [PWRF] [ 0.942176] PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 0.943246] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 0.944828] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 0.946988] PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.948045] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 0.950092] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.974435] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 0.976467] Non-volatile memory driver v1.3 [ 0.977507] rdac: device handler registered [ 0.978388] hp_sw: device handler registered [ 0.979223] emc: device handler registered [ 0.980066] alua: device handler registered [ 0.980990] libphy: Fixed MDIO Bus: probed [ 0.981882] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.983117] ehci-pci: EHCI PCI platform driver [ 0.983981] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.985220] ohci-pci: OHCI PCI platform driver [ 0.986095] uhci_hcd: USB Universal Host Controller Interface driver [ 0.988240] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.989337] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.990737] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.991801] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c040 [ 0.992978] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 4.18 [ 0.994538] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.995891] usb usb1: Product: UHCI Host Controller [ 0.996851] usb usb1: Manufacturer: Linux 4.18.0-348.2.1.el8_lustre.x86_64 uhci_hcd [ 0.998307] usb usb1: SerialNumber: 0000:00:01.2 [ 0.999360] hub 1-0:1.0: USB hub found [ 1.000149] hub 1-0:1.0: 2 ports detected [ 1.001160] usbcore: registered new interface driver usbserial_generic [ 1.002427] usbserial: USB Serial support registered for generic [ 1.003646] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 1.006198] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.007196] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.008333] mousedev: PS/2 mouse device common for all mice [ 1.009858] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 1.011749] rtc_cmos 00:00: RTC can wake from S4 [ 1.013525] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 1.015621] rtc_cmos 00:00: registered as rtc0 [ 1.016582] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 1.017785] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 1.019419] intel_pstate: CPU model not supported [ 1.020735] hid: raw HID events driver (C) Jiri Kosina [ 1.021793] usbcore: registered new interface driver usbhid [ 1.022874] usbhid: USB HID core driver [ 1.023689] drop_monitor: Initializing network drop monitor service [ 1.024956] Initializing XFRM netlink socket [ 1.025977] NET: Registered protocol family 10 [ 1.027196] Segment Routing with IPv6 [ 1.027960] NET: Registered protocol family 17 [ 1.028914] mpls_gso: MPLS GSO support [ 1.029959] AVX version of gcm_enc/dec engaged. [ 1.030868] AES CTR mode by8 optimization enabled [ 1.082125] sched_clock: Marking stable (1082109794, 0)->(1468921600, -386811806) [ 1.083762] registered taskstats version 1 [ 1.084620] Loading compiled-in X.509 certificates [ 1.126638] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: b362844b5793fe6dbb885a47b0325cbd0bd85c15' [ 1.129041] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80' [ 1.131577] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8' [ 1.133555] zswap: loaded using pool lzo/zbud [ 1.134510] page_owner is disabled [ 1.144688] Key type big_key registered [ 1.150147] Key type encrypted registered [ 1.150948] ima: No TPM chip found, activating TPM-bypass! [ 1.152012] ima: Allocated hash algorithm: sha256 [ 1.152918] ima: No architecture policies found [ 1.153824] evm: Initialising EVM extended attributes: [ 1.154805] evm: security.selinux [ 1.155469] evm: security.ima [ 1.156067] evm: security.capability [ 1.156766] evm: HMAC attrs: 0x1 [ 1.157799] rtc_cmos 00:00: setting system clock to 2022-05-02 16:35:42 UTC (1651509342) [ 1.161186] Freeing unused decrypted memory: 2036K [ 1.163173] Freeing unused kernel memory: 2492K [ 1.173027] Write protecting the kernel read-only data: 22528k [ 1.176209] Freeing unused kernel memory: 2012K [ 1.177452] Freeing unused kernel memory: 320K [ 1.191171] systemd[1]: systemd 239 (239-51.el8_5.2) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 1.195285] systemd[1]: Detected virtualization kvm. [ 1.196260] systemd[1]: Detected architecture x86-64. [ 1.197241] systemd[1]: Running in initial RAM disk. [ 1.207335] systemd[1]: Set hostname to . [ 1.273915] random: systemd: uninitialized urandom read (16 bytes read) [ 1.275225] systemd[1]: Reached target Slices. [ 1.277237] random: systemd: uninitialized urandom read (16 bytes read) [ 1.278571] systemd[1]: Listening on udev Kernel Socket. [ 1.280754] random: systemd: uninitialized urandom read (16 bytes read) [ 1.282133] systemd[1]: Listening on Journal Socket. [ 1.285780] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 1.288996] systemd[1]: Listening on Journal Socket (/dev/log). [ 1.562006] random: fast init done [ 1.657854] virtio_blk virtio1: [vda] 209715200 512-byte logical blocks (107 GB/100 GiB) [ 1.684120] vda: detected capacity change from 0 to 107374182400 [ 1.725451] libata version 3.00 loaded. [ 1.727681] ata_piix 0000:00:01.1: version 2.13 [ 1.728877] vda: vda1 vda2 vda3 [ 1.733221] scsi host0: ata_piix [ 1.740498] scsi host1: ata_piix [ 1.741271] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14 [ 1.742699] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15 [ 1.842136] PM: Image not found (code -22) [ 3.184882] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ 4.088592] printk: systemd: 14 output lines suppressed due to ratelimiting [ 5.346095] SELinux: Disabled at runtime. [ 5.380056] audit: type=1404 audit(1651509346.722:2): enforcing=0 old_enforcing=0 auid=4294967295 ses=4294967295 enabled=0 old-enabled=1 lsm=selinux res=1 [ 5.522094] systemd[1]: systemd 239 (239-51.el8_5.2) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.526211] systemd[1]: Detected virtualization kvm. [ 5.527191] systemd[1]: Detected architecture x86-64. [ 5.545660] systemd[1]: Set hostname to . [ 7.698794] systemd[1]: systemd-journald.service: Succeeded. [ 7.700815] systemd[1]: initrd-switch-root.service: Succeeded. [ 7.702359] systemd[1]: Stopped Switch Root. [ 7.704540] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 7.706549] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 7.708320] systemd[1]: Stopped Journal Service. [ 7.802044] Adding 2753532k swap on /dev/vda2. Priority:-2 extents:1 across:2753532k FS [ 7.927043] EXT4-fs (vda1): re-mounted. Opts: (null) [ 8.816814] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 8.882565] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 9.349178] RPC: Registered named UNIX socket transport module. [ 9.350367] RPC: Registered udp transport module. [ 9.351295] RPC: Registered tcp transport module. [ 9.352219] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 13.623896] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 13.741023] random: crng init done [ 13.741761] random: 7 urandom warning(s) missed due to ratelimiting [ 14.816575] FS-Cache: Loaded [ 14.881045] FS-Cache: Netfs 'nfs' registered for caching [ 14.890666] Key type dns_resolver registered [ 15.080196] NFS: Registering the id_resolver key type [ 15.081254] Key type id_resolver registered [ 15.082121] Key type id_legacy registered [ 38.267587] alg: No test for pkcs1pad(rsa,sha1) (pkcs1pad(rsa-generic,sha1)) [ 38.285623] PKCS7: Message signed outside of X.509 validity window [12335.857035] libcfs: loading out-of-tree module taints kernel. [12335.858826] libcfs: module verification failed: signature and/or required key missing - tainting kernel [12335.865835] LNet: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 [12335.871652] alg: No test for adler32 (adler32-zlib) [12336.624497] Key type ._llcrypt registered [12336.625363] Key type .llcrypt registered [12336.651552] Lustre: Lustre: Build Version: 2.15.0_RC3 [12336.688398] LNet: Added LNI 10.240.43.63@tcp [8/256/0/180] [12336.690014] LNet: Accept secure, port 988 [12336.805691] LDISKFS-fs (vda3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12338.094721] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 300-900 [12340.313526] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [16558.953616] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre [16561.030802] Lustre: DEBUG MARKER: trevis-87vm4.trevis.whamcloud.com: executing check_logdir /tmp/test_logs/2022-05-02/211139 [16562.121654] Lustre: DEBUG MARKER: trevis-87vm4.trevis.whamcloud.com: executing yml_node [16562.996270] Lustre: DEBUG MARKER: Client: 2.15.0.RC3 [16563.494391] Lustre: DEBUG MARKER: MDS: 2.15.0.RC3 [16563.988222] Lustre: DEBUG MARKER: OSS: 2.15.0.RC3 [16564.299696] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Mon May 2 21:11:47 UTC 2022 [16565.589169] Lustre: DEBUG MARKER: excepting tests: 42a 42b 42c 407 312 817 411 [16565.876167] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 115 135 136 230d 300o [16567.012445] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre [16570.275845] Lustre: DEBUG MARKER: Using TIMEOUT=100 [16574.640605] Lustre: DEBUG MARKER: trevis-87vm4.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [16575.552111] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ============================= 21:11:58 (1651525918) [16576.750636] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ======================================================================================= 21:11:59 (1651525919) [16577.928820] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ===================== 21:12:00 (1651525920) [16579.114814] Lustre: DEBUG MARKER: == sanity test 0d: check export proc ======================================================================================= 21:12:01 (1651525921) [16581.703070] Lustre: DEBUG MARKER: == sanity test 0e: Enable DNE MDT balancing for mkdir in the ROOT ========================================================== 21:12:04 (1651525924) [16581.992574] Lustre: DEBUG MARKER: SKIP: sanity test_0e We need at least 2 MDTs for this test [16582.311491] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ================== 21:12:05 (1651525925) [16583.535466] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ======== 21:12:06 (1651525926) [16584.774543] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ========= 21:12:07 (1651525927) [16586.243593] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ========================================================== 21:12:09 (1651525929) [16587.498480] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ========================================================== 21:12:10 (1651525930) [16588.729725] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ============================================================ 21:12:11 (1651525931) [16589.926309] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ============================================================ 21:12:12 (1651525932) [16591.161092] Lustre: DEBUG MARKER: == sanity test 6e: touch+chgrp ; runas -u 500 -g 500 chgrp (should return error) ========================================================== 21:12:14 (1651525934) [16592.387899] Lustre: DEBUG MARKER: == sanity test 6g: verify new dir in sgid dir inherits group ========================================================== 21:12:15 (1651525935) [16593.589224] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../ (should return error) ========================================================== 21:12:16 (1651525936) [16594.797315] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ============================================================== 21:12:17 (1651525937) [16595.980030] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 =============================================================== 21:12:18 (1651525938) [16597.196382] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ================================================================= 21:12:20 (1651525940) [16598.391930] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ========================================================================== 21:12:21 (1651525941) [16600.797745] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ================================================================ 21:12:23 (1651525943) [16602.016641] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ====================================================================== 21:12:24 (1651525944) [16603.267405] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ================================================================== 21:12:26 (1651525946) [16604.471594] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ================================================================== 21:12:27 (1651525947) [16605.670245] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ================================================================= 21:12:28 (1651525948) [16608.092525] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ==================================================================== 21:12:30 (1651525950) [16609.281344] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ===== 21:12:32 (1651525952) [16610.473949] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ====== 21:12:33 (1651525953) [16611.686116] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) == 21:12:34 (1651525954) [16614.088744] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ========================================================== 21:12:35 (1651525955) [16615.301436] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ============ 21:12:38 (1651525958) [16616.525854] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================================================== 21:12:39 (1651525959) [16617.741974] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ========================================================== 21:12:40 (1651525960) [16618.982466] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================================================== 21:12:41 (1651525961) [16620.401923] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ========================================================== 21:12:43 (1651525963) [16621.823507] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink (should return error) ========================================================== 21:12:44 (1651525964) [16623.243636] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled == 21:12:46 (1651525966) [16624.515386] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ========================================================== 21:12:47 (1651525967) [16625.753654] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================================================== 21:12:48 (1651525968) [16649.956947] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [16649.959665] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [16649.963080] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b158550b9efc3 to 0x7e7b158550bbc12e [16649.965506] Lustre: MGC10.240.43.61@tcp: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [16655.247957] Lustre: lustre-OST0001: deleting orphan objects from 0x0:47 to 0x0:65 [16657.981499] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ========================================================== 21:13:20 (1651526000) [16658.273809] Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs [16658.588304] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature == 21:13:21 (1651526001) [16682.210805] Lustre: 39099:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651526018/real 1651526018] req@000000004ab68f56 x1731745996105024/t0(0) o400->MGC10.240.43.61@tcp@10.240.43.61@tcp:26/25 lens 224/224 e 0 to 1 dl 1651526025 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:2.0' [16682.210847] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [16682.216237] Lustre: 39099:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16682.216252] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [16682.225666] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [16682.227519] Lustre: Skipped 1 previous similar message [16682.228528] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b158550bbc12e to 0x7e7b158550bc880d [16683.901057] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [16684.989542] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16685.315491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [16688.077859] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ======================================================================================== 21:13:50 (1651526030) [16689.268310] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 ===================================================================== 21:13:52 (1651526032) [16690.574808] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ======================================================================== 21:13:53 (1651526033) [16693.041421] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ============================================================ 21:13:55 (1651526035) [16694.228983] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ======================================================================== 21:13:57 (1651526037) [16696.622410] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =============== 21:13:58 (1651526038) [16697.848975] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ================ 21:14:00 (1651526040) [16700.530260] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user === 21:14:03 (1651526043) [16703.031049] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ============= 21:14:05 (1651526045) [16704.222866] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check ======================= 21:14:07 (1651526047) [16705.437254] Lustre: DEBUG MARKER: == sanity test 23c: O_APPEND size checks for tiny writes ========================================================== 21:14:08 (1651526048) [16707.472065] Lustre: DEBUG MARKER: == sanity test 23d: file offset is correct after appending writes ========================================================== 21:14:10 (1651526050) [16708.675983] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target === 21:14:11 (1651526051) [16709.886065] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target ======= 21:14:12 (1651526052) [16711.122799] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================================== 21:14:13 (1651526053) [16712.617443] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target == 21:14:15 (1651526055) [16713.876672] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ================================================================ 21:14:16 (1651526056) [16715.106729] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ============================================================== 21:14:17 (1651526057) [16716.357826] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ================================================================ 21:14:19 (1651526059) [16717.568617] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ========================================================== 21:14:20 (1651526060) [16719.033300] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================================================== 21:14:21 (1651526061) [16720.259495] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ====================================================================================== 21:14:23 (1651526063) [16721.456965] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ================================================================= 21:14:24 (1651526064) [16722.666188] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ================================================================================== 21:14:25 (1651526065) [16723.852288] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself =================================================================== 21:14:26 (1651526066) [16725.047118] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ========================================================== 21:14:27 (1651526067) [16726.236245] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split === 21:14:29 (1651526069) [16766.162407] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ========================================================== 21:15:09 (1651526109) [16768.588867] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b ============================================================= 21:15:11 (1651526111) [16770.135599] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b =============================================================== 21:15:12 (1651526112) [16771.334665] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c =========================================================== 21:15:14 (1651526114) [16772.555720] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a =========================================================== 21:15:15 (1651526115) [16773.772957] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =================== 21:15:16 (1651526116) [16775.016495] Lustre: DEBUG MARKER: == sanity test 24v: list large directory (test hash collision, b=17560) ========================================================== 21:15:17 (1651526117) [16867.799620] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb ======= 21:16:50 (1651526210) [16869.058308] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ================ 21:16:51 (1651526211) [16869.344660] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [16869.663236] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ========================================================== 21:16:52 (1651526212) [16869.954332] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [16870.275545] Lustre: DEBUG MARKER: == sanity test 24z: cross-MDT rename is done as cp ======= 21:16:53 (1651526213) [16870.566142] Lustre: DEBUG MARKER: SKIP: sanity test_24z needs >= 2 MDTs [16870.881501] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ========================================================== 21:16:53 (1651526213) [16882.372755] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================================================== 21:17:05 (1651526225) [16882.666531] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [16882.981329] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ============== 21:17:05 (1651526225) [16883.270650] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [16883.586849] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ================ 21:17:06 (1651526226) [16883.874840] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [16884.190041] Lustre: DEBUG MARKER: == sanity test 24F: hash order vs readdir (LU-11330) ===== 21:17:07 (1651526227) [16884.476668] Lustre: DEBUG MARKER: SKIP: sanity test_24F needs >= 2 MDTs [16884.789588] Lustre: DEBUG MARKER: == sanity test 24G: migrate symlink in rename ============ 21:17:07 (1651526227) [16885.079606] Lustre: DEBUG MARKER: SKIP: sanity test_24G needs >= 2 MDTs [16885.399519] Lustre: DEBUG MARKER: == sanity test 24H: repeat FLD_QUERY rpc ================= 21:17:08 (1651526228) [16885.691535] Lustre: DEBUG MARKER: SKIP: sanity test_24H needs >= 2 MDTs [16886.006299] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory ========================================================================= 21:17:08 (1651526228) [16887.280092] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory ========================================================================= 21:17:10 (1651526230) [16888.483478] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ================================================================================= 21:17:11 (1651526231) [16889.695591] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ================================================================ 21:17:12 (1651526232) [16890.908487] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ==================== 21:17:13 (1651526233) [16892.120325] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ========================================================== 21:17:14 (1651526234) [16893.325773] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ========================================================== 21:17:16 (1651526236) [16894.525306] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ========================================================== 21:17:17 (1651526237) [16895.858044] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ====================== 21:17:18 (1651526238) [16897.096128] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file == 21:17:19 (1651526239) [16898.334057] Lustre: DEBUG MARKER: == sanity test 27ca: one stripe on specified OST ========= 21:17:21 (1651526241) [16899.648428] Lustre: DEBUG MARKER: == sanity test 27cb: two stripes on specified OSTs ======= 21:17:22 (1651526242) [16900.940645] Lustre: DEBUG MARKER: == sanity test 27cc: two stripes on the same OST ========= 21:17:23 (1651526243) [16902.293701] Lustre: DEBUG MARKER: == sanity test 27cd: four stripes on two OSTs ============ 21:17:25 (1651526245) [16903.609441] Lustre: DEBUG MARKER: == sanity test 27ce: more stripes than OSTs with -o ====== 21:17:26 (1651526246) [16904.932354] Lustre: DEBUG MARKER: == sanity test 27cf: 'setstripe -o' on inactive OSTs should return error ========================================================== 21:17:27 (1651526247) [16908.079226] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ==== 21:17:30 (1651526250) [16909.296277] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ========================================================== 21:17:32 (1651526252) [16910.531132] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ========================================================== 21:17:33 (1651526253) [16911.782885] Lustre: DEBUG MARKER: == sanity test 27g: /usr/bin/lfs getstripe with no objects ========================================================== 21:17:34 (1651526254) [16912.988223] Lustre: DEBUG MARKER: == sanity test 27ga: /usr/bin/lfs getstripe with missing file (should return error) ========================================================== 21:17:35 (1651526255) [16914.211573] Lustre: DEBUG MARKER: == sanity test 27i: /usr/bin/lfs getstripe with some objects ========================================================== 21:17:37 (1651526257) [16915.412019] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) ========================================================== 21:17:38 (1651526258) [16916.634864] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ========================================================== 21:17:39 (1651526259) [16917.869818] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ========================================================== 21:17:40 (1651526260) [16919.059189] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [16919.372870] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ====== 21:17:42 (1651526262) [16937.385072] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ========================================================== 21:18:00 (1651526280) [16942.873383] Lustre: *** cfs_fail_loc=215, val=0*** [16944.951668] Lustre: *** cfs_fail_loc=215, val=1*** [16950.071353] Lustre: *** cfs_fail_loc=215, val=1*** [16956.718700] Lustre: DEBUG MARKER: == sanity test 27oo: don't let few threads to reserve too many objects ========================================================== 21:18:19 (1651526299) [16988.068128] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs ========================================================== 21:18:50 (1651526330) [17015.123828] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ========================================================== 21:19:18 (1651526358) [17021.747571] Lustre: *** cfs_fail_loc=215, val=1*** [17026.867242] Lustre: *** cfs_fail_loc=215, val=1*** [17034.710726] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) =========================================================== 21:19:37 (1651526377) [17051.507821] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ========================================================== 21:19:54 (1651526394) [17052.710417] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ========================================================== 21:19:55 (1651526395) [17054.082014] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================================== 21:19:56 (1651526396) [17064.497594] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ===== 21:20:07 (1651526407) [17067.825194] Lustre: *** cfs_fail_loc=215, val=1*** [17132.822133] Lustre: DEBUG MARKER: == sanity test 27w: check /usr/bin/lfs setstripe -S and getstrip -d options ========================================================== 21:21:15 (1651526475) [17134.051065] Lustre: DEBUG MARKER: == sanity test 27wa: check /usr/bin/lfs setstripe -c -i options ========================================================== 21:21:16 (1651526476) [17135.288627] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded == 21:21:18 (1651526478) [17148.321455] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ========================================================== 21:21:31 (1651526491) [17159.985337] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 10.240.43.61@tcp) reconnecting [17159.992445] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1271 to 0x0:1313 [17171.375403] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ========================================================== 21:21:54 (1651526514) [17177.011971] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [17177.324438] Lustre: DEBUG MARKER: FID seq 0x200000bd1, oid 0x1cd96 ver 0x0 [17177.611661] Lustre: DEBUG MARKER: LOV seq 0x200000bd1, oid 0x1cd96, count: 1 [17177.907601] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:377/0x179 seq:0 [17186.033438] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [17186.346173] Lustre: DEBUG MARKER: FID seq 0x200000bd1, oid 0x1cd98 ver 0x0 [17186.632551] Lustre: DEBUG MARKER: LOV seq 0x200000bd1, oid 0x1cd98, count: 2 [17186.944336] Lustre: DEBUG MARKER: want: stripe:0 ost:1 oid:1315/0x523 seq:0 [17187.604783] Lustre: DEBUG MARKER: want: stripe:1 ost:0 oid:378/0x17a seq:0 [17195.729855] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ========================================================== 21:22:18 (1651526538) [17198.354780] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim ========================================================== 21:22:21 (1651526541) [17199.601471] Lustre: DEBUG MARKER: == sanity test 27Ca: check full striping across all OSTs ========================================================== 21:22:22 (1651526542) [17200.874461] Lustre: DEBUG MARKER: == sanity test 27Cb: more stripes than OSTs with -C ====== 21:22:23 (1651526543) [17202.185783] Lustre: DEBUG MARKER: == sanity test 27Cc: fewer stripes than OSTs does not set overstriping ========================================================== 21:22:25 (1651526545) [17203.499904] Lustre: DEBUG MARKER: == sanity test 27Cd: test maximum stripe count =========== 21:22:26 (1651526546) [17203.993298] Lustre: DEBUG MARKER: SKIP: sanity test_27Cd ea_inode feature disabled [17204.314826] Lustre: DEBUG MARKER: == sanity test 27Ce: test pool with overstriping ========= 21:22:27 (1651526547) [17216.730385] Lustre: DEBUG MARKER: == sanity test 27Cf: test default inheritance with overstriping ========================================================== 21:22:39 (1651526559) [17218.081403] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API ============ 21:22:40 (1651526560) [17231.721896] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ========================================================== 21:22:54 (1651526574) [17232.957986] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================================================== 21:22:55 (1651526575) [17234.272664] Lustre: Failing over lustre-OST0001 [17234.434489] LustreError: 71707:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [17234.555672] Lustre: server umount lustre-OST0001 complete [17235.020029] device-mapper: uevent: version 1.0.3 [17235.021490] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com [17240.069505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17240.185722] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [17240.189439] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [17241.194209] Lustre: DEBUG MARKER: trevis-87vm4.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [17241.294018] Lustre: lustre-OST0001: Will be in recovery for at least 2:30, or until 2 clients reconnect [17241.434902] Lustre: lustre-OST0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [17241.439438] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1337 to 0x0:1377 [17243.750181] Lustre: DEBUG MARKER: == sanity test 27G: Clear OST pool from stripe =========== 21:23:06 (1651526586) [17254.997837] Lustre: DEBUG MARKER: == sanity test 27H: Set specific OSTs stripe ============= 21:23:17 (1651526597) [17255.291746] Lustre: DEBUG MARKER: SKIP: sanity test_27H needs >= 3 OSTs [17255.610347] Lustre: DEBUG MARKER: == sanity test 27I: check that root dir striping does not break parent dir one ========================================================== 21:23:18 (1651526598) [17266.800265] Lustre: DEBUG MARKER: == sanity test 27J: basic ops on file with foreign LOV === 21:23:29 (1651526609) [17268.211763] Lustre: DEBUG MARKER: == sanity test 27K: basic ops on dir with foreign LMV ==== 21:23:31 (1651526611) [17269.758422] Lustre: DEBUG MARKER: == sanity test 27L: lfs pool_list gives correct pool name ========================================================== 21:23:32 (1651526612) [17276.374250] Lustre: DEBUG MARKER: == sanity test 27M: test O_APPEND striping =============== 21:23:39 (1651526619) [17293.554685] Lustre: DEBUG MARKER: == sanity test 27N: lctl pool_list on separate MGS gives correct pool name ========================================================== 21:23:56 (1651526636) [17293.863116] Lustre: DEBUG MARKER: SKIP: sanity test_27N needs separate MGS/MDT [17294.187672] Lustre: DEBUG MARKER: == sanity test 27O: basic ops on foreign file of symlink type ========================================================== 21:23:57 (1651526637) [17295.713394] Lustre: DEBUG MARKER: == sanity test 27P: basic ops on foreign dir of foreign_symlink type ========================================================== 21:23:58 (1651526638) [17297.300239] Lustre: DEBUG MARKER: == sanity test 27Q: llapi_file_get_stripe() works on symlinks ========================================================== 21:24:00 (1651526640) [17298.560039] Lustre: DEBUG MARKER: == sanity test 27R: test max_stripecount limitation when stripe count is set to -1 ========================================================== 21:24:01 (1651526641) [17300.483045] Lustre: DEBUG MARKER: == sanity test 27S: don't deactivate OSP on network issue ========================================================== 21:24:03 (1651526643) [17303.927114] Lustre: DEBUG MARKER: == sanity test 27T: no eio on close on partial write due to enosp ========================================================== 21:24:06 (1651526646) [17305.510255] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ====================================================================== 21:24:08 (1651526648) [17306.827086] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ====================================================================================== 21:24:09 (1651526649) [17308.422824] Lustre: DEBUG MARKER: first d29 [17308.732074] Lustre: DEBUG MARKER: second d29 [17310.247843] Lustre: DEBUG MARKER: done [17311.471849] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ======================================================================== 21:24:14 (1651526654) [17312.724913] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root ===================================================================== 21:24:15 (1651526655) [17313.980972] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ============================================================== 21:24:16 (1651526656) [17315.264231] Lustre: DEBUG MARKER: == sanity test 30d: execute binary from Lustre while clear locks ========================================================== 21:24:18 (1651526658) [17347.363905] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ============================================================================================ 21:24:50 (1651526690) [17348.574673] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ================================================================= 21:24:51 (1651526691) [17349.804839] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ======================================================================= 21:24:52 (1651526692) [17351.054775] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory =================================================================================== 21:24:53 (1651526693) [17352.282239] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory ========================================================================= 21:24:55 (1651526695) [17353.549056] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file ============================================================= 21:24:56 (1651526696) [17360.857475] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link================== 21:25:03 (1651526703) [17362.110169] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child========================================================================= 21:25:04 (1651526704) [17363.339560] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent========================================================================= 21:25:06 (1651526706) [17364.586998] Lustre: DEBUG MARKER: == sanity test 31j: link for directory==================== 21:25:07 (1651526707) [17365.843542] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================================================= 21:25:08 (1651526708) [17367.084451] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================================================= 21:25:09 (1651526709) [17368.317668] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ==== 21:25:11 (1651526711) [17369.548085] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================================== 21:25:12 (1651526712) [17377.328139] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ===== 21:25:20 (1651526720) [17377.621984] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [17377.945017] Lustre: DEBUG MARKER: == sanity test 31q: create striped directory on specific MDTs ========================================================== 21:25:20 (1651526720) [17378.244237] Lustre: DEBUG MARKER: SKIP: sanity test_31q needs >= 3 MDTs [17378.566647] Lustre: DEBUG MARKER: == sanity test 31r: open-rename(replace) race ============ 21:25:21 (1651526721) [17382.836915] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. =============================================================================== 21:25:25 (1651526725) [17384.292567] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. =============================================================================== 21:25:27 (1651526727) [17385.659842] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir =================================================================== 21:25:28 (1651526728) [17387.046684] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================================== 21:25:29 (1651526729) [17388.423253] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ========================================================== 21:25:31 (1651526731) [17389.682646] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ========================================================== 21:25:32 (1651526732) [17390.925747] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ========================================================== 21:25:33 (1651526733) [17392.162348] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ========================================================== 21:25:35 (1651526735) [17393.421310] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file ===================================================================== 21:25:36 (1651526736) [17394.772335] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file ===================================================================== 21:25:37 (1651526737) [17396.137862] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ================================================================== 21:25:39 (1651526739) [17397.516230] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ================================================================== 21:25:40 (1651526740) [17398.886116] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ================================================================ 21:25:41 (1651526741) [17400.129191] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ================================================================ 21:25:43 (1651526743) [17401.369048] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ========================================================== 21:25:44 (1651526744) [17402.621503] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ========================================================== 21:25:45 (1651526745) [17402.916207] Lustre: DEBUG MARKER: 32p_1 [17403.215376] Lustre: DEBUG MARKER: 32p_2 [17403.528447] Lustre: DEBUG MARKER: 32p_3 [17403.825945] Lustre: DEBUG MARKER: 32p_4 [17404.128973] Lustre: DEBUG MARKER: 32p_5 [17404.420879] Lustre: DEBUG MARKER: 32p_6 [17404.731583] Lustre: DEBUG MARKER: 32p_7 [17405.031638] Lustre: DEBUG MARKER: 32p_8 [17405.331785] Lustre: DEBUG MARKER: 32p_9 [17405.632756] Lustre: DEBUG MARKER: 32p_10 [17406.842987] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================================================== 21:25:49 (1651526749) [17408.315209] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ========================================================== 21:25:51 (1651526751) [17409.668141] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ========================================================== 21:25:52 (1651526752) [17409.984208] Lustre: DEBUG MARKER: 33_1 [17410.282055] Lustre: DEBUG MARKER: 33_2 [17411.496442] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ========================================================== 21:25:54 (1651526754) [17412.757319] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) ========================================================== 21:25:55 (1651526755) [17414.010394] Lustre: DEBUG MARKER: == sanity test 33c: test write_bytes stats =============== 21:25:56 (1651526756) [17415.596260] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ========================================================== 21:25:58 (1651526758) [17415.886784] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [17416.200899] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ========================================================== 21:25:59 (1651526759) [17416.496244] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [17416.818230] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ========================================================== 21:25:59 (1651526759) [17417.112235] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [17417.438704] Lustre: DEBUG MARKER: == sanity test 33g: nonroot user create already existing root created file ========================================================== 21:26:00 (1651526760) [17418.667772] Lustre: DEBUG MARKER: == sanity test 33h: temp file is located on the same MDT as target ========================================================== 21:26:01 (1651526761) [17418.957835] Lustre: DEBUG MARKER: SKIP: sanity test_33h needs >= 2 MDTs [17419.278122] Lustre: DEBUG MARKER: == sanity test 33i: striped directory can be accessed when one MDT is down ========================================================== 21:26:02 (1651526762) [17419.570498] Lustre: DEBUG MARKER: SKIP: sanity test_33i needs >= 2 MDTs [17419.899136] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened ===================================================================== 21:26:02 (1651526762) [17421.149854] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects =============================================================== 21:26:04 (1651526764) [17423.003445] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ======================================================================== 21:26:05 (1651526765) [17424.236200] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ======================================================================================= 21:26:07 (1651526767) [17425.463928] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ============================================================ 21:26:08 (1651526768) [17426.698771] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ================================================================= 21:26:09 (1651526769) [17427.944356] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file ========================================================================================= 21:26:10 (1651526770) [17429.196857] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ========================================================== 21:26:12 (1651526772) [17432.758256] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ========================================================== 21:26:15 (1651526775) [17434.034963] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) ======= 21:26:16 (1651526776) [17435.254863] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ======== 21:26:18 (1651526778) [17436.476927] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================================== 21:26:19 (1651526779) [17437.740956] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) ========================================================== 21:26:20 (1651526780) [17438.962082] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ========================================================== 21:26:21 (1651526781) [17440.190175] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ==================================================================== 21:26:23 (1651526783) [17442.477703] Lustre: DEBUG MARKER: == sanity test 36g: FMD cache expiry =============================================================================== 21:26:25 (1651526785) [17566.310041] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ==================================================================== 21:28:29 (1651526909) [17568.577713] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ==== 21:28:31 (1651526911) [17568.877223] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [17569.198837] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ============================================================= 21:28:32 (1651526912) [17570.461209] Lustre: DEBUG MARKER: == sanity test 39a: mtime changed on create ============== 21:28:33 (1651526913) [17573.692793] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ================================================================ 21:28:36 (1651526916) [17576.061602] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename ===================================================================================== 21:28:38 (1651526918) [17579.315930] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ======================================================================================== 21:28:42 (1651526922) [17580.568091] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ================================================================================== 21:28:43 (1651526923) [17581.804437] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat =========================================================================== 21:28:44 (1651526924) [17585.083093] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat ========================================================================================= 21:28:47 (1651526927) [17588.335111] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ======================================================================= 21:28:51 (1651526931) [17590.597924] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ======================================================================================== 21:28:53 (1651526933) [17592.865632] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ================================================================================= 21:28:55 (1651526935) [17597.328556] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ================================================================================== 21:29:00 (1651526940) [17599.703999] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update ===================================================================================== 21:29:02 (1651526942) [17608.931720] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ===== 21:29:11 (1651526951) [17612.187980] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ====== 21:29:15 (1651526955) [17624.031134] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ========================================================== 21:29:26 (1651526966) [17625.328009] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ================================================================== 21:29:28 (1651526968) [17625.634852] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [17625.955555] Lustre: DEBUG MARKER: == sanity test 39r: lazy atime update on OST ============= 21:29:28 (1651526968) [17639.391702] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime =========== 21:29:42 (1651526982) [17640.623164] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ======================================================================= 21:29:43 (1651526983) [17641.857334] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat =============================================================================== 21:29:44 (1651526984) [17644.239070] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [17644.530880] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [17644.828760] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [17645.145982] Lustre: DEBUG MARKER: == sanity test 42d: test complete truncate of file with cached dirty data ========================================================== 21:29:48 (1651526988) [17646.932429] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ========================================================== 21:29:49 (1651526989) [17678.305174] Lustre: DEBUG MARKER: == sanity test 43A: execution of file opened for write should return -ETXTBSY ========================================================== 21:30:21 (1651527021) [17680.635670] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================================================== 21:30:23 (1651527023) [17682.910659] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY ========================================================== 21:30:25 (1651527025) [17685.163451] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre =========== 21:30:28 (1651527028) [17686.443658] Lustre: DEBUG MARKER: == sanity test 44A: zero length read from a sparse stripe ========================================================== 21:30:29 (1651527029) [17687.693170] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite ========================================================================================= 21:30:30 (1651527030) [17689.181882] Lustre: DEBUG MARKER: == sanity test 45: osc io page accounting ====================================================================================== 21:30:32 (1651527032) [17690.701370] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ========================================================================== 21:30:33 (1651527033) [17692.065210] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)=========================================================== 21:30:34 (1651527034) [17693.372823] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)=========================================================== 21:30:36 (1651527036) [17694.626541] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ========================================================== 21:30:37 (1651527037) [17695.887236] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ========================================================== 21:30:38 (1651527038) [17697.170308] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================================================== 21:30:40 (1651527040) [17698.424259] Lustre: DEBUG MARKER: == sanity test 48f: non-zero nlink dir unlink won't LBUG() ========================================================== 21:30:41 (1651527041) [17698.719646] Lustre: DEBUG MARKER: SKIP: sanity test_48f needs >= 2 MDTs [17699.044057] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent ========================================================== 21:30:41 (1651527041) [17701.206938] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks ========================================================================= 21:30:44 (1651527044) [17702.450497] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ============================================================ 21:30:45 (1651527045) [17705.875269] Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit on create, verify unlink ========================================================== 21:30:48 (1651527048) [17772.254062] Lustre: DEBUG MARKER: == sanity test 51d: check object distribution ============ 21:31:55 (1651527115) [17772.748984] Lustre: DEBUG MARKER: SKIP: sanity test_51d needs >= 3 OSTs [17773.065211] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =============== 21:31:55 (1651527115) [17827.737915] Lustre: DEBUG MARKER: == sanity test 51f: check many open files limit ========== 21:32:49 (1651527169) [17924.276885] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ========================================================== 21:34:27 (1651527267) [17926.218078] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ================================================================= 21:34:29 (1651527269) [17927.562398] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ============================================================== 21:34:30 (1651527270) [17930.189197] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ==================================================================================== 21:34:33 (1651527273) [17931.500384] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ================================================================================ 21:34:34 (1651527274) [17932.849453] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre =============================================================================== 21:34:35 (1651527275) [17934.836184] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ================================================================================ 21:34:37 (1651527277) [17936.466314] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre ================================================================================ 21:34:39 (1651527279) [17937.761762] Lustre: DEBUG MARKER: == sanity test 56a: check /usr/bin/lfs getstripe ========= 21:34:40 (1651527280) [17939.174676] Lustre: DEBUG MARKER: == sanity test 56b: check /usr/bin/lfs getdirstripe ====== 21:34:42 (1651527282) [17940.451342] Lustre: DEBUG MARKER: == sanity test 56c: check 'lfs df' showing device status ========================================================== 21:34:43 (1651527283) [17963.422518] Lustre: DEBUG MARKER: == sanity test 56d: 'lfs df -v' prints only configured devices ========================================================== 21:35:06 (1651527306) [17964.661198] Lustre: DEBUG MARKER: == sanity test 56e: 'lfs df' Handle non LustreFS [17965.984323] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ================= 21:35:08 (1651527308) [17967.341053] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =============== 21:35:10 (1651527310) [17968.607065] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ========================================================== 21:35:11 (1651527311) [17969.842062] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =============== 21:35:12 (1651527312) [17971.163470] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =============== 21:35:14 (1651527314) [17972.395968] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =============== 21:35:15 (1651527315) [17973.732127] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =============== 21:35:16 (1651527316) [17974.971325] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =============== 21:35:17 (1651527317) [17976.207603] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files == 21:35:19 (1651527319) [17977.722706] Lustre: DEBUG MARKER: == sanity test 56ob: check lfs find -atime -mtime -ctime with units ========================================================== 21:35:20 (1651527320) [17982.374446] Lustre: DEBUG MARKER: == sanity test 56oc: check lfs find -newerXY work ======== 21:35:24 (1651527324) [18045.041919] Lustre: DEBUG MARKER: == sanity test 56od: check lfs find -btime with units ==== 21:36:27 (1651527387) [18051.885946] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid ======= 21:36:34 (1651527394) [18053.262518] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid ======= 21:36:36 (1651527396) [18054.653690] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works =========== 21:36:37 (1651527397) [18056.781718] Lustre: DEBUG MARKER: == sanity test 56ra: check lfs find -size -lazy works for data on OSTs ========================================================== 21:36:39 (1651527399) [18058.824097] Lustre: DEBUG MARKER: == sanity test 56rb: check lfs find --size --ost/--mdt works ========================================================== 21:36:41 (1651527401) [18061.320157] Lustre: DEBUG MARKER: == sanity test 56rc: check lfs find --mdt-count/--mdt-hash works ========================================================== 21:36:44 (1651527404) [18061.627292] Lustre: DEBUG MARKER: SKIP: sanity test_56rc needs at least 2 MDTs [18061.958299] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works === 21:36:44 (1651527404) [18063.389706] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ==== 21:36:46 (1651527406) [18065.270312] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works === 21:36:48 (1651527408) [18066.711603] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -m match with lfs getstripe -m' ========================================================== 21:36:49 (1651527409) [18068.134936] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================================== 21:36:51 (1651527411) [18079.150188] Lustre: DEBUG MARKER: == sanity test 56wb: check lfs_migrate pool support ====== 21:37:02 (1651527422) [18091.982057] Lustre: DEBUG MARKER: == sanity test 56wc: check unrecognized options for lfs_migrate are passed through ========================================================== 21:37:14 (1651527434) [18093.977655] Lustre: DEBUG MARKER: == sanity test 56wd: check lfs_migrate --rsync and --no-rsync work ========================================================== 21:37:16 (1651527436) [18095.350200] Lustre: DEBUG MARKER: == sanity test 56we: check lfs_migrate --non-direct|-D support ========================================================== 21:37:18 (1651527438) [18096.803544] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ================ 21:37:19 (1651527439) [18098.223404] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support ======= 21:37:21 (1651527441) [18099.642811] Lustre: DEBUG MARKER: == sanity test 56xb: lfs migration hard link support ===== 21:37:22 (1651527442) [18126.158712] Lustre: DEBUG MARKER: == sanity test 56xc: lfs migration autostripe ============ 21:37:49 (1651527469) [18127.905531] Lustre: DEBUG MARKER: == sanity test 56xd: check lfs_migrate --yaml and --copy support ========================================================== 21:37:50 (1651527470) [18131.229399] Lustre: DEBUG MARKER: == sanity test 56xe: migrate a composite layout file ===== 21:37:54 (1651527474) [18134.796204] Lustre: DEBUG MARKER: == sanity test 56xf: FID is not lost during migration of a composite layout file ========================================================== 21:37:57 (1651527477) [18137.929238] Lustre: DEBUG MARKER: == sanity test 56xg: lfs migrate pool support ============ 21:38:00 (1651527480) [18174.716401] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released =========== 21:38:37 (1651527517) [18176.008178] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================================== 21:38:38 (1651527518) [18178.716870] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir === 21:38:41 (1651527521) [18179.018903] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [18179.347539] Lustre: DEBUG MARKER: == sanity test 56ab: lfs find --blocks =================== 21:38:42 (1651527522) [18181.526669] Lustre: DEBUG MARKER: == sanity test 56aca: check lfs find -perm with octal representation ========================================================== 21:38:44 (1651527524) [18184.718877] Lustre: DEBUG MARKER: == sanity test 56acb: check lfs find -perm with symbolic representation ========================================================== 21:38:47 (1651527527) [18186.465818] Lustre: DEBUG MARKER: == sanity test 56acc: check parsing error for lfs find -perm ========================================================== 21:38:49 (1651527529) [18187.723234] Lustre: DEBUG MARKER: == sanity test 56ba: test lfs find --component-end, -start, -count, and -flags ========================================================== 21:38:50 (1651527530) [18189.484477] Lustre: DEBUG MARKER: == sanity test 56ca: check lfs find --mirror-count|-N and --mirror-state ========================================================== 21:38:52 (1651527532) [18191.484934] Lustre: DEBUG MARKER: == sanity test 56da: test lfs find with long paths ======= 21:38:54 (1651527534) [18194.351340] Lustre: DEBUG MARKER: == sanity test 56ea: test lfs find -printf option ======== 21:38:57 (1651527537) [18208.994053] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ============================================================ 21:39:11 (1651527551) [18210.874090] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes ============================================================= 21:39:13 (1651527553) [18215.217799] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ======================================================================== 21:39:18 (1651527558) [18216.495854] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async =================================================================== 21:39:19 (1651527559) [18232.350945] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 21:39:35 (1651527575) [18232.843093] Lustre: DEBUG MARKER: test_60 run 21060 - from kernel mode [18259.086500] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [18266.253721] Lustre: 39100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651527602/real 1651527602] req@000000006731397d x1731745996219648/t0(0) o400->MGC10.240.43.61@tcp@10.240.43.61@tcp:26/25 lens 224/224 e 0 to 1 dl 1651527609 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' [18266.259177] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [18270.082554] Lustre: lustre-OST0001: deleting orphan objects from 0x0:53038 to 0x0:53057 [18272.399344] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b158551451baf to 0x7e7b15855234af71 [18272.402084] Lustre: MGC10.240.43.61@tcp: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [18272.403815] Lustre: Skipped 1 previous similar message [18272.494493] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 21:40:15 (1651527615) [18273.961688] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 21:40:16 (1651527616) [18277.520216] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [18281.699313] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 21:40:24 (1651527624) [18282.970493] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 21:40:25 (1651527625) [18284.636453] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 21:40:27 (1651527627) [18287.033111] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 21:40:29 (1651527629) [18313.952471] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 21:40:55 (1651527655) [18314.254493] Lustre: DEBUG MARKER: SKIP: sanity test_60h Need at least 2 MDTs [18314.584832] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [18314.933706] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 21:40:57 (1651527657) [18316.269360] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 21:40:59 (1651527659) [18317.507787] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 21:41:00 (1651527660) [18381.051419] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 21:42:03 (1651527723) [18386.483527] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 21:42:09 (1651527729) [18388.075334] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [18388.394659] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 21:42:11 (1651527731) [18390.005370] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 21:42:12 (1651527732) [18413.677902] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 21:42:36 (1651527756) [18417.075498] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 21:42:40 (1651527760) [18420.641130] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 21:42:43 (1651527763) [18458.342852] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 21:43:21 (1651527801) [18468.250566] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 21:43:31 (1651527811) [18494.333153] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [18494.679682] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [18497.362030] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 21:44:00 (1651527840) [18498.626241] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 21:44:01 (1651527841) [18499.886573] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 21:44:02 (1651527842) [18502.352469] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 21:44:05 (1651527845) [18503.622070] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 21:44:06 (1651527846) [18504.895615] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 21:44:07 (1651527847) [18506.171286] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 21:44:09 (1651527849) [18507.452324] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 21:44:10 (1651527850) [18508.723097] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 21:44:11 (1651527851) [18510.506252] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 21:44:13 (1651527853) [18512.801674] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 21:44:15 (1651527855) [18513.410764] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 10.240.43.61@tcp) reconnecting [18513.413184] Lustre: lustre-OST0001: deleting orphan objects from 0x0:55565 to 0x0:55585 [18522.441344] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 10.240.43.61@tcp) reconnecting [18522.447161] Lustre: lustre-OST0001: deleting orphan objects from 0x0:56588 to 0x0:56609 [18524.768981] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 21:44:27 (1651527867) [18526.040318] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 21:44:28 (1651527868) [18527.304001] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 21:44:30 (1651527870) [18532.650062] Lustre: DEBUG MARKER: SKIP: sanity test_65n needs >= 2 MDTs [18540.465877] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 21:44:43 (1651527883) [18543.116663] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 21:44:46 (1651527886) [18545.869908] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [18546.193234] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 21:44:49 (1651527889) [18547.537436] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 21:44:50 (1651527890) [18548.982012] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 21:44:51 (1651527891) [18576.284095] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 21:45:19 (1651527919) [18577.545622] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 21:45:20 (1651527920) [18580.006428] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 21:45:22 (1651527922) [18581.274206] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 21:45:24 (1651527924) [18584.143106] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 21:45:27 (1651527927) [18588.118068] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 21:45:31 (1651527931) [18589.468905] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 21:45:32 (1651527932) [18590.885729] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [18592.325677] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [18593.732828] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [18595.134582] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [18596.863995] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [18598.275117] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [18599.685330] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [18601.085063] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [18602.331895] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 21:45:45 (1651527945) [18612.840567] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 21:45:55 (1651527955) [18612.882476] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.240.43.64@tcp inode [0x200002345:0xb27:0x0] object 0x0:56746 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [18615.423312] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.240.43.64@tcp inode [0x200002345:0xb27:0x0] object 0x0:56746 extent [0-4194303], client returned csum f5a99216 (type 4), server csum de2bf73e (type 4) [18616.789444] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 21:45:59 (1651527959) [18617.118646] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [18667.162473] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [18716.857336] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [18765.015374] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [18813.162604] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [18862.281711] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [18911.389302] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [18959.621780] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [18960.887545] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 21:51:43 (1651528303) [18965.197467] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 21:51:48 (1651528308) [18971.135108] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 21:51:54 (1651528314) [18971.452374] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [18971.762143] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [18972.913609] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [18994.444278] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in IDLE state after 21 sec [18995.563353] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [18995.858571] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in FULL state after 0 sec [18996.178913] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [18997.329839] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19019.871954] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in IDLE state after 22 sec [19021.005709] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19021.318805] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in FULL state after 0 sec [19021.629422] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [19022.774739] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19044.304946] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in IDLE state after 21 sec [19045.686899] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19045.992302] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in FULL state after 0 sec [19046.320446] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [19047.465494] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19070.021829] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in IDLE state after 22 sec [19071.149423] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19071.443286] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in FULL state after 0 sec [19071.759069] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [19072.893563] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19095.438807] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in IDLE state after 22 sec [19097.766439] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19098.060406] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in FULL state after 0 sec [19098.376739] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [19099.620456] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19120.141373] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in IDLE state after 20 sec [19121.268211] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19122.774032] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in FULL state after 0 sec [19123.090593] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [19124.238540] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19144.761846] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in IDLE state after 20 sec [19146.045179] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19146.348889] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in FULL state after 0 sec [19147.591188] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [19147.925548] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 21:54:50 (1651528490) [19149.166119] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 21:54:52 (1651528492) [19149.554575] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [19149.868016] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [19150.174574] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [19150.503853] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [19151.772731] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [19152.108080] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 21:54:55 (1651528495) [19154.469068] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 21:54:57 (1651528497) [19158.042459] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 21:55:01 (1651528501) [19171.103664] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 21:55:14 (1651528514) [19173.516193] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 21:55:16 (1651528516) [19175.118788] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 21:55:18 (1651528518) [19176.721773] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 21:55:19 (1651528519) [19180.019513] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ===================================================================== 21:55:22 (1651528522) [19181.947184] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 21:55:24 (1651528524) [19188.315051] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 21:55:31 (1651528531) [19191.010463] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 21:55:33 (1651528533) [19195.200057] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 21:55:38 (1651528538) [19215.763793] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 21:55:58 (1651528558) [19219.857691] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 21:56:02 (1651528562) [19223.788875] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 21:56:06 (1651528566) [19229.162144] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 21:56:12 (1651528572) [19233.798401] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 21:56:16 (1651528576) [19235.402239] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 21:56:18 (1651528578) [19238.444147] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 21:56:21 (1651528581) [19240.111351] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 21:56:23 (1651528583) [19242.231738] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 21:56:25 (1651528585) [19243.614441] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 21:56:26 (1651528586) [19246.893055] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 21:56:29 (1651528589) [19248.959310] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 21:56:31 (1651528591) [19249.369158] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [19249.684354] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [19249.997511] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [19250.314952] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [19251.585901] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 21:56:34 (1651528594) [19252.097243] Lustre: DEBUG MARKER: SKIP: sanity test_102ha ea_inode feature disabled [19252.434023] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 21:56:35 (1651528595) [19253.754601] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 21:56:36 (1651528596) [19255.749738] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 21:56:38 (1651528598) [19257.097047] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 21:56:40 (1651528600) [19258.409848] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 21:56:41 (1651528601) [19259.684739] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 21:56:42 (1651528602) [19261.106650] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 21:56:44 (1651528604) [19262.397929] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 21:56:45 (1651528605) [19263.781408] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 21:56:46 (1651528606) [19265.122087] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 21:56:48 (1651528608) [19266.485380] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 21:56:49 (1651528609) [19267.840264] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 21:56:50 (1651528610) [19283.274199] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 21:57:06 (1651528626) [19295.053000] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 21:57:18 (1651528638) [19296.416800] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 21:57:19 (1651528639) [19296.931896] Lustre: DEBUG MARKER: SKIP: sanity test_103e ea_inode feature disabled [19297.261968] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 21:57:20 (1651528640) [19297.774003] Lustre: DEBUG MARKER: SKIP: sanity test_103f ea_inode feature disabled [19298.103596] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 21:57:21 (1651528641) [19299.295335] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid 200 [19299.587508] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95f15c4ae800.ost_server_uuid in FULL state after 0 sec [19300.877510] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 21:57:23 (1651528643) [19302.172878] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 21:57:25 (1651528645) [19302.472898] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [19302.814102] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 21:57:25 (1651528645) [19304.099560] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 21:57:27 (1651528647) [19305.379112] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 21:57:28 (1651528648) [19306.656652] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 21:57:29 (1651528649) [19318.036151] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 21:57:41 (1651528661) [19319.323079] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 21:57:42 (1651528662) [19320.598635] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 21:57:43 (1651528663) [19323.069558] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 21:57:46 (1651528666) [19324.387468] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [19324.708032] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 21:57:47 (1651528667) [19338.257238] Lustre: DEBUG MARKER: SKIP: sanity test_116a too much free space in OST0, skip [19338.589449] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 21:58:01 (1651528681) [19340.858340] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 21:58:03 (1651528683) [19342.160512] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 21:58:05 (1651528685) [19343.628835] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 21:58:06 (1651528686) [19345.690414] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ==================================================================== 21:58:08 (1651528688) [19349.508964] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ==================================================================== 21:58:12 (1651528692) [19349.918076] LustreError: 77221:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 25000ms [19374.986169] LustreError: 77221:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [19376.585495] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 21:58:39 (1651528719) [19378.044406] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 21:58:41 (1651528721) [19379.515603] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 21:58:42 (1651528722) [19391.650168] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 21:58:54 (1651528734) [19392.016752] Lustre: *** cfs_fail_loc=20e, val=0*** [19393.044908] Lustre: *** cfs_fail_loc=20e, val=0*** [19395.092821] Lustre: *** cfs_fail_loc=20e, val=0*** [19399.773687] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 21:59:02 (1651528742) [19400.139692] Lustre: *** cfs_fail_loc=220, val=0*** [19401.689090] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 21:59:04 (1651528744) [19404.531213] Lustre: *** cfs_fail_loc=20e, val=0*** [19404.532272] Lustre: Skipped 4 previous similar messages [19410.081553] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 21:59:13 (1651528753) [19411.386853] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 21:59:14 (1651528754) [19412.796847] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 21:59:15 (1651528755) [19417.323426] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 21:59:20 (1651528760) [19418.670345] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 21:59:21 (1651528761) [19420.235811] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 21:59:23 (1651528763) [19421.657090] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ========================================================== 21:59:24 (1651528764) [19423.074488] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [19426.375072] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 21:59:29 (1651528769) [19428.125983] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 21:59:31 (1651528771) [19429.850618] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 21:59:32 (1651528772) [19432.938004] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 21:59:35 (1651528775) [19435.872376] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 21:59:38 (1651528778) [19444.663569] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 21:59:47 (1651528787) [19453.465843] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 21:59:56 (1651528796) [19473.420157] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 22:00:16 (1651528816) [19474.764414] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 22:00:17 (1651528817) [19475.332964] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 0 sec [19475.759310] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [19478.605481] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 1 sec [19479.578906] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 1 sec [19497.700149] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 11 sec [19503.483716] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 5 sec [19503.804492] Lustre: DEBUG MARKER: ls -l done [19509.085239] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity/: 5 seconds [19509.396299] Lustre: DEBUG MARKER: rm done [19510.847066] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 22:00:53 (1651528853) [19511.616482] Lustre: DEBUG MARKER: statx -l 100 files without statahead: 0 sec [19512.034304] Lustre: DEBUG MARKER: statx -l 100 files with statahead: 1 sec [19515.918338] Lustre: DEBUG MARKER: statx -l 1000 files without statahead: 1 sec [19516.802246] Lustre: DEBUG MARKER: statx -l 1000 files with statahead: 0 sec [19534.316512] Lustre: DEBUG MARKER: statx -l 10000 files without statahead: 11 sec [19539.247140] Lustre: DEBUG MARKER: statx -l 10000 files with statahead: 4 sec [19539.569205] Lustre: DEBUG MARKER: statx -l done [19544.967040] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ab.sanity/: 5 seconds [19545.287677] Lustre: DEBUG MARKER: rm done [19546.881139] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 22:01:29 (1651528889) [19547.444419] Lustre: DEBUG MARKER: statx -c %n [19547.803742] Lustre: DEBUG MARKER: statx -c %n [19549.999420] Lustre: DEBUG MARKER: statx -c %n [19550.532222] Lustre: DEBUG MARKER: statx -c %n [19562.038778] Lustre: DEBUG MARKER: statx -c %n [19563.921713] Lustre: DEBUG MARKER: statx -c %n [19672.018975] Lustre: DEBUG MARKER: statx -c %n [19688.692100] Lustre: DEBUG MARKER: statx -c %n [19689.013541] Lustre: DEBUG MARKER: statx -c %n [19739.501051] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ac.sanity/: 50 seconds [19739.809013] Lustre: DEBUG MARKER: rm done [19740.265733] Lustre: DEBUG MARKER: statx --cached=always -D 100 files without statahead: 0 sec [19740.608799] Lustre: DEBUG MARKER: statx --cached=always -D 100 files with statahead: 0 sec [19742.382296] Lustre: DEBUG MARKER: statx --cached=always -D 1000 files without statahead: 0 sec [19742.734823] Lustre: DEBUG MARKER: statx --cached=always -D 1000 files with statahead: 0 sec [19749.772286] Lustre: DEBUG MARKER: statx --cached=always -D 10000 files without statahead: 0 sec [19750.131616] Lustre: DEBUG MARKER: statx --cached=always -D 10000 files with statahead: 0 sec [19810.403390] Lustre: DEBUG MARKER: statx --cached=always -D 100000 files without statahead: 0 sec [19810.862071] Lustre: DEBUG MARKER: statx --cached=always -D 100000 files with statahead: 0 sec [19811.188460] Lustre: DEBUG MARKER: statx --cached=always -D done [19904.670291] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ac.sanity/: 93 seconds [19904.978381] Lustre: DEBUG MARKER: rm done [19906.277137] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 22:07:29 (1651529249) [19908.657115] Lustre: DEBUG MARKER: ls done [19910.485253] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 22:07:33 (1651529253) [19910.797162] Lustre: DEBUG MARKER: SKIP: sanity test_123c needs >= 2 MDTs [19911.128198] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 22:07:34 (1651529254) [19911.446178] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [19914.844518] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff95f15c4ae800 [19915.152547] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff95f15c4ae800 [19915.466823] Lustre: DEBUG MARKER: LRU=2003 [19915.778354] Lustre: DEBUG MARKER: LIMIT=133199 [19916.092004] Lustre: DEBUG MARKER: LVF=7979900 [19916.398968] Lustre: DEBUG MARKER: OLD_LVF=100 [19916.700876] Lustre: DEBUG MARKER: Sleep 50 sec [19967.069127] Lustre: DEBUG MARKER: Dropped 848 locks in 50s [19967.382839] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [19971.487772] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 22:08:34 (1651529314) [19975.213249] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [19982.065832] Lustre: DEBUG MARKER: ls -la time: 7 seconds [19982.377881] Lustre: DEBUG MARKER: lru_size = 200 [19989.651969] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [19995.634445] Lustre: DEBUG MARKER: ls -la time: 5 seconds [19995.954479] Lustre: DEBUG MARKER: lru_size = 4006 [19996.267638] Lustre: DEBUG MARKER: ls -la is 28% faster with lru resize enabled [19999.642008] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 22:09:02 (1651529342) [20021.193055] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize diasbaled ========================================================== 22:09:24 (1651529364) [20042.778052] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 22:09:45 (1651529385) [20045.379433] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 22:09:47 (1651529387) [20047.848383] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 22:09:50 (1651529390) [20049.301796] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 22:09:52 (1651529392) [20050.632562] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [20053.800028] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 22:09:56 (1651529396) [20055.076204] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 22:09:58 (1651529398) [20062.672460] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 22:10:05 (1651529405) [20063.995874] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 22:10:07 (1651529407) [20065.336413] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 22:10:08 (1651529408) [20066.679287] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 22:10:09 (1651529409) [20066.984684] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [20067.324148] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 22:10:10 (1651529410) [20074.052928] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 22:10:17 (1651529417) [20075.341728] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 22:10:18 (1651529418) [20079.161402] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 22:10:20 (1651529420) [20080.474039] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 22:10:23 (1651529423) [20081.795208] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 22:10:24 (1651529424) [20083.074383] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 22:10:26 (1651529426) [20084.380294] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 22:10:27 (1651529427) [20085.905736] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 22:10:28 (1651529428) [20090.789637] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 22:10:33 (1651529433) [20099.745007] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 22:10:42 (1651529442) [20117.351907] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 22:11:00 (1651529460) [20122.097721] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 22:11:05 (1651529465) [20124.493709] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 22:11:07 (1651529467) [20147.753962] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [20147.757164] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [20147.762417] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [20147.765574] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b15855234af71 to 0x7e7b158553dd5e7e [20149.941256] Lustre: lustre-OST0001: deleting orphan objects from 0x0:179426 to 0x0:179489 [20151.019490] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [20151.337270] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [20172.839688] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [20180.006901] Lustre: 39100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651529516/real 1651529516] req@0000000036a3a267 x1731745997078912/t0(0) o400->MGC10.240.43.61@tcp@10.240.43.61@tcp:26/25 lens 224/224 e 0 to 1 dl 1651529523 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:2.0' [20180.012376] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [20184.036437] LustreError: 163139:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [20184.038260] LustreError: 163139:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 6 previous similar messages [20184.184635] Lustre: server umount lustre-OST0001 complete [20185.585953] device-mapper: core: cleaned up [20200.710309] device-mapper: uevent: version 1.0.3 [20200.711647] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com [20202.336719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20202.485115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 300-900 [20203.641734] Lustre: DEBUG MARKER: trevis-87vm4.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [20208.202023] Lustre: DEBUG MARKER: Using TIMEOUT=100 [20209.289502] Lustre: lustre-OST0001: deleting orphan objects from 0x0:179426 to 0x0:179521 [20215.221898] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 22:12:38 (1651529558) [20278.817998] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [20292.128838] Lustre: 39100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651529628/real 1651529628] req@00000000946e402f x1731745997081920/t0(0) o400->MGC10.240.43.61@tcp@10.240.43.61@tcp:26/25 lens 224/224 e 0 to 1 dl 1651529635 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:2.0' [20292.134280] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [20300.159852] LustreError: 165085:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [20300.161660] LustreError: 165085:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 5 previous similar messages [20300.194241] Lustre: server umount lustre-OST0001 complete [20302.785536] device-mapper: core: cleaned up [20317.819486] device-mapper: uevent: version 1.0.3 [20317.820591] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com [20319.378761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20319.529361] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 300-900 [20320.691211] Lustre: DEBUG MARKER: trevis-87vm4.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [20324.512783] Lustre: DEBUG MARKER: Using TIMEOUT=100 [20327.554849] Lustre: lustre-OST0001: deleting orphan objects from 0x0:179426 to 0x0:179553 [20332.722400] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 22:14:35 (1651529675) [20453.522592] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 22:16:36 (1651529796) [20468.372930] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 22:16:51 (1651529811) [20572.861610] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [20573.172249] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [20574.700338] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 22:18:36 (1651529916) [20578.074851] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 22:18:41 (1651529921) [20589.565413] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 22:18:52 (1651529932) [20601.206662] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 22:19:04 (1651529944) [20616.689268] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 22:19:19 (1651529959) [20626.416533] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 22:19:29 (1651529969) [20629.466351] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 22:19:32 (1651529972) [20643.955323] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 22:19:47 (1651529987) [20657.474923] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 22:20:00 (1651530000) [20667.541333] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 22:20:10 (1651530010) [20670.894746] bash (170894): drop_caches: 1 [20677.998925] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 22:20:19 (1651530019) [20679.347624] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 22:20:22 (1651530022) [20680.686317] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 22:20:23 (1651530023) [20681.986576] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 22:20:25 (1651530025) [20683.298460] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 22:20:26 (1651530026) [20684.917362] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 22:20:27 (1651530027) [20685.221952] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [20685.556077] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 22:20:28 (1651530028) [20686.871612] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 22:20:29 (1651530029) [20689.621826] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 22:20:32 (1651530032) [20690.988080] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 22:20:34 (1651530034) [20692.434166] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 22:20:35 (1651530035) [20728.205851] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 22:21:11 (1651530071) [20730.874612] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 22:21:13 (1651530073) [20733.547766] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 22:21:16 (1651530076) [20736.195769] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 22:21:19 (1651530079) [20738.837002] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 22:21:21 (1651530081) [20751.528189] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 22:21:34 (1651530094) [20761.408491] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 22:21:44 (1651530104) [20771.283879] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 22:21:54 (1651530114) [20782.423738] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 22:22:05 (1651530125) [20784.185809] Lustre: DEBUG MARKER: Turn on read and write cache [20785.029254] Lustre: DEBUG MARKER: Write data and read it back. [20785.480144] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [20786.338729] Lustre: DEBUG MARKER: cache hits: before: 8237, after: 8240 [20786.647124] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [20788.439543] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8243 [20788.744386] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [20789.589964] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [20790.449450] Lustre: DEBUG MARKER: cache hits:: before: 8243, after: 8246 [20790.759508] Lustre: DEBUG MARKER: Write data and read it back. [20791.061335] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [20791.928429] Lustre: DEBUG MARKER: cache hits:: before: 8246, after: 8249 [20792.243336] Lustre: DEBUG MARKER: Turn off read and write cache [20793.083588] Lustre: DEBUG MARKER: Write data and read it back [20793.391795] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [20794.259203] Lustre: DEBUG MARKER: cache hits:: before: 8249, after: 8249 [20794.561185] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [20795.402304] Lustre: DEBUG MARKER: Write data and read it back [20795.715451] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [20797.778744] Lustre: DEBUG MARKER: cache hits:: before: 8249, after: 8249 [20798.087366] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [20798.938456] Lustre: DEBUG MARKER: cache hits:: before: 8249, after: 8252 [20800.589866] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 22:22:23 (1651530143) [20815.525805] Lustre: lustre-OST0001: deleting orphan objects from 0x0:180386 to 0x0:180417 [20818.923340] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 22:22:41 (1651530161) [20822.856257] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 22:22:45 (1651530165) [20824.068664] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [20824.068671] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [20824.071678] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b158553dd6298 to 0x7e7b158553e60190 [20824.076671] Lustre: MGC10.240.43.61@tcp: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [20824.078423] Lustre: Skipped 1 previous similar message [20825.882055] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 22:22:48 (1651530168) [20826.220064] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [20826.556895] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 22:22:49 (1651530169) [20829.365625] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 22:22:52 (1651530172) [20830.877403] Lustre: DEBUG MARKER: 1651530173: creating first files [20846.491593] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 22:23:09 (1651530189) [20855.903073] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 22:23:18 (1651530198) [20874.241930] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [20881.409181] Lustre: 39100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651530217/real 1651530217] req@0000000086faad8f x1731745997090688/t0(0) o400->MGC10.240.43.61@tcp@10.240.43.61@tcp:26/25 lens 224/224 e 0 to 1 dl 1651530224 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:2.0' [20881.414651] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [20881.421735] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [20881.423575] Lustre: Skipped 1 previous similar message [20881.424633] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b158553e60190 to 0x7e7b158553e6062f [20882.069325] Lustre: lustre-OST0001: deleting orphan objects from 0x0:180386 to 0x0:180449 [20885.938415] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 22:23:49 (1651530229) [20892.432313] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 22:23:55 (1651530235) [20892.828199] Lustre: DEBUG MARKER: sanity test_160j: @@@@@@ FAIL: mount_client on /mnt/lustre2 failed [20898.864485] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 22:24:01 (1651530241) [20909.037431] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 22:24:12 (1651530252) [20914.706226] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 22:24:16 (1651530256) [20920.787430] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 22:24:23 (1651530263) [21055.272675] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 22:26:38 (1651530398) [21060.730234] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 22:26:43 (1651530403) [21083.283964] Lustre: lustre-OST0001: deleting orphan objects from 0x0:180452 to 0x0:180481 [21083.638753] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [21083.638757] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [21083.644585] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b158553e6062f to 0x7e7b158554297508 [21083.647200] Lustre: MGC10.240.43.61@tcp: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [21083.649617] Lustre: Skipped 1 previous similar message [21084.925018] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 22:27:07 (1651530427) [21087.051102] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records acceptance-small.sh acl aiocp auster badarea_io cfg check_fallocate check_fhandle_syscalls checkfiemap checkstat chownmany clientapi compile.sh conf-sanity.sh create_foreign_dir create_foreign [21093.983427] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 22:27:17 (1651530437) [21098.076031] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 22:27:21 (1651530441) [21098.384397] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [21098.716096] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 22:27:21 (1651530441) [21102.676456] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 22:27:24 (1651530444) [21109.628527] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 22:27:31 (1651530451) [21111.041350] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 22:27:34 (1651530454) [21111.347411] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [21111.686404] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 22:27:34 (1651530454) [21116.143945] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 22:27:39 (1651530459) [21128.001083] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 22:27:51 (1651530471) [21154.645875] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 22:28:17 (1651530497) [21169.667092] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 22:28:32 (1651530512) [21195.335113] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 22:28:58 (1651530538) [21195.650939] Lustre: DEBUG MARKER: SKIP: sanity test_165e needs >= 2 MDTs [21195.982680] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 22:28:59 (1651530539) [21207.327071] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 22:29:10 (1651530550) [21207.638322] Lustre: DEBUG MARKER: creating a 10 Mb file [21208.519911] Lustre: DEBUG MARKER: starting reads [21208.870205] Lustre: DEBUG MARKER: truncating the file [21209.198946] Lustre: DEBUG MARKER: killing dd [21209.507753] Lustre: DEBUG MARKER: removing the temporary file [21210.809660] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log =============================================================================== 22:29:13 (1651530553) [21212.235728] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 22:29:15 (1651530555) [21216.610970] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 22:29:19 (1651530559) [21218.788744] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 22:29:21 (1651530561) [21223.269506] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 22:29:26 (1651530566) [21235.393887] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 22:29:38 (1651530578) [21242.108191] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================================================== 22:29:45 (1651530585) [21255.429887] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 22:29:58 (1651530598) [21257.260347] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 22:30:00 (1651530600) [21259.197436] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 22:30:02 (1651530602) [21260.563397] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 22:30:03 (1651530603) [21265.024415] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 22:30:08 (1651530608) [21267.374303] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 22:30:10 (1651530610) [21269.589494] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 22:30:12 (1651530612) [21271.169177] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 22:30:14 (1651530614) [21273.513876] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 22:30:16 (1651530616) [21278.043756] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 22:30:21 (1651530621) [21279.816420] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 22:30:22 (1651530622) [21281.516075] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 22:30:24 (1651530624) [21294.978790] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 22:30:38 (1651530638) [21296.302295] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 22:30:39 (1651530639) [21298.837627] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 22:30:41 (1651530641) [21300.152281] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 22:30:43 (1651530643) [21301.467653] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 22:30:44 (1651530644) [21302.785912] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 22:30:45 (1651530645) [21304.100881] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 22:30:47 (1651530647) [21305.431235] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 22:30:48 (1651530648) [21307.934983] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 22:30:51 (1651530651) [21311.541584] Lustre: DEBUG MARKER: Test: /usr/bin/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [21311.853605] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.30092 [21312.364514] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [21312.677860] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.18146 [21313.193437] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [21313.504162] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.18732 [21314.015191] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [21314.329191] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.28006 [21314.846727] Lustre: DEBUG MARKER: Test: /usr/bin/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [21315.164470] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.27427 [21315.681192] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [21315.997348] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.24038 [21316.702398] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [21317.024628] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.24248 [21317.706324] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [21318.176412] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.12231 [21318.700473] Lustre: DEBUG MARKER: Test: /lib64/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [21319.019523] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.20139 [21319.725366] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [21320.037431] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.28169 [21320.745827] Lustre: DEBUG MARKER: Test: /usr/bin/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [21321.065410] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.12645 [21324.149227] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [21324.474302] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.trevis-87vm5.tre [21324.989010] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [21325.304055] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.trevis-87vm5.E [21325.831103] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [21326.146599] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.trevis-87vm5.tre [21328.894932] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 22:31:11 (1651530671) [21330.881972] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 22:31:13 (1651530673) [21332.182770] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 22:31:15 (1651530675) [21333.492286] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 22:31:16 (1651530676) [21334.830511] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 22:31:17 (1651530677) [21336.171792] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 22:31:19 (1651530679) [21361.639377] Lustre: 39099:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651530698/real 1651530698] req@00000000c683c211 x1731745997102976/t0(0) o400->MGC10.240.43.61@tcp@10.240.43.61@tcp:26/25 lens 224/224 e 0 to 1 dl 1651530705 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:2.0' [21361.644852] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [21361.648284] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b158554297508 to 0x7e7b1585543ae2b6 [21361.650766] Lustre: MGC10.240.43.61@tcp: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [21365.307890] Lustre: lustre-OST0001: deleting orphan objects from 0x0:187586 to 0x0:187617 [21366.390761] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21366.717789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [21388.262785] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b1585543ae2b6 to 0x7e7b1585543ae3ff [21388.265718] Lustre: MGC10.240.43.61@tcp: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [21391.193351] Lustre: lustre-OST0001: deleting orphan objects from 0x0:187586 to 0x0:187649 [21392.265716] Lustre: DEBUG MARKER: trevis-87vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21392.578066] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [21395.099295] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 22:32:18 (1651530738) [21398.501379] Lustre: 39099:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651530698/real 1651530698] req@0000000009c4c564 x1731745997103040/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.240.43.61@tcp:12/10 lens 224/224 e 0 to 1 dl 1651530741 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:2.0' [21398.506926] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [21398.512235] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [21403.104886] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 22:32:26 (1651530746) [21406.444637] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 22:32:29 (1651530749) [21408.073785] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 22:32:31 (1651530751) [21419.526675] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 22:32:42 (1651530762) [21422.205740] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 22:32:45 (1651530765) [21423.612644] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 22:32:46 (1651530766) [21424.101037] Lustre: 39099:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651530724/real 1651530724] req@000000009b569a7a x1731745997104000/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.240.43.61@tcp:12/10 lens 224/224 e 0 to 1 dl 1651530767 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' [21427.959604] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ========================================================== 22:32:51 (1651530771) [21430.034094] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 22:32:53 (1651530773) [21430.342426] Lustre: DEBUG MARKER: creating a 10 Mb file [21431.184154] Lustre: DEBUG MARKER: starting reads [21431.516160] Lustre: DEBUG MARKER: truncating the file [21431.850059] Lustre: DEBUG MARKER: killing dd [21432.158898] Lustre: DEBUG MARKER: removing the temporary file [21433.480379] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 22:32:56 (1651530776) [21434.868216] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 22:32:57 (1651530777) [21445.038328] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 22:33:08 (1651530788) [21452.249951] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 22:33:15 (1651530795) [21453.665268] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 22:33:16 (1651530796) [21456.222918] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 22:33:19 (1651530799) [21457.588770] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 22:33:20 (1651530800) [21463.600400] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 22:33:26 (1651530806) [21469.584215] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 22:33:32 (1651530812) [21487.007017] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 22:33:50 (1651530830) [21590.836757] Lustre: DEBUG MARKER: == sanity test 225a: Metadata survey sanity with zero-stripe ========================================================== 22:35:33 (1651530933) [21600.477965] Lustre: DEBUG MARKER: == sanity test 225b: Metadata survey sanity with stripe_count = 1 ========================================================== 22:35:43 (1651530943) [21610.208619] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 22:35:53 (1651530953) [21611.650801] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 22:35:54 (1651530954) [21611.973774] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [21612.305698] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 22:35:55 (1651530955) [21612.629291] Lustre: DEBUG MARKER: SKIP: sanity test_226c needs >= 2 MDTs [21613.006226] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 22:35:56 (1651530956) [21614.461912] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 22:35:57 (1651530957) [21643.867964] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 22:36:26 (1651530986) [21673.100567] Lustre: lustre-OST0001: deleting orphan objects from 0x0:199343 to 0x0:199617 [21683.550875] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 22:37:06 (1651531026) [21685.206170] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [21685.208830] LustreError: Skipped 1 previous similar message [21685.208892] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [21685.213203] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b1585543ae3ff to 0x7e7b15855451b7bd [21685.216928] Lustre: MGC10.240.43.61@tcp: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [21685.218834] Lustre: Skipped 1 previous similar message [21723.078444] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 22:37:46 (1651531066) [21724.692424] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 22:37:47 (1651531067) [21725.005195] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [21725.349287] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 22:37:48 (1651531068) [21725.656167] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [21726.004877] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 22:37:49 (1651531069) [21726.309852] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [21726.641267] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [21726.973368] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 22:37:50 (1651531070) [21727.280727] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [21727.619486] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 22:37:50 (1651531070) [21727.931017] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [21728.270138] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 22:37:51 (1651531071) [21728.607218] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [21728.946477] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 22:37:52 (1651531072) [21729.258052] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [21729.597088] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 22:37:52 (1651531072) [21729.910784] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [21730.256434] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 22:37:53 (1651531073) [21730.571781] Lustre: DEBUG MARKER: SKIP: sanity test_230j needs >= 2 MDTs [21730.917186] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 22:37:54 (1651531074) [21731.230587] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [21731.573939] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 22:37:54 (1651531074) [21731.883884] Lustre: DEBUG MARKER: SKIP: sanity test_230l needs >= 2 MDTs [21732.230271] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 22:37:55 (1651531075) [21732.542560] Lustre: DEBUG MARKER: SKIP: sanity test_230m needs >= 2 MDTs [21732.888953] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 22:37:55 (1651531075) [21733.200715] Lustre: DEBUG MARKER: SKIP: sanity test_230n needs >= 2 MDTs [21733.540641] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 22:37:56 (1651531076) [21733.857699] Lustre: DEBUG MARKER: SKIP: sanity test_230o needs >= 2 MDTs [21734.202691] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 22:37:57 (1651531077) [21734.510878] Lustre: DEBUG MARKER: SKIP: sanity test_230p needs >= 2 MDTs [21734.857191] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 22:37:57 (1651531077) [21735.167795] Lustre: DEBUG MARKER: SKIP: sanity test_230q needs >= 2 MDTs [21735.508898] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 22:37:58 (1651531078) [21735.817551] Lustre: DEBUG MARKER: SKIP: sanity test_230r needs >= 2 MDTs [21736.172528] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 22:37:59 (1651531079) [21738.459086] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 22:38:01 (1651531081) [21738.765427] Lustre: DEBUG MARKER: SKIP: sanity test_230t needs >= 2 MDTs [21739.576433] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 22:38:02 (1651531082) [21741.094429] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [21741.430273] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 22:38:04 (1651531084) [21741.744747] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [21742.095698] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 22:38:05 (1651531085) [21742.403411] Lustre: DEBUG MARKER: SKIP: sanity test_230w needs >= 2 MDTs [21742.749376] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 22:38:05 (1651531085) [21745.918328] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 22:38:09 (1651531089) [21749.523165] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 22:38:12 (1651531092) [21756.449713] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 22:38:19 (1651531099) [21762.802310] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 22:38:25 (1651531105) [21764.126230] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 22:38:27 (1651531107) [21765.456433] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 22:38:28 (1651531108) [21766.834288] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 22:38:29 (1651531109) [21770.171859] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 22:38:33 (1651531113) [21771.563151] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 22:38:34 (1651531114) [21772.948843] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 22:38:36 (1651531116) [21782.590903] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 22:38:45 (1651531125) [21794.723556] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 22:38:57 (1651531137) [21799.447611] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 22:39:02 (1651531142) [21799.761395] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [21800.108301] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 22:39:03 (1651531143) [21813.445478] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 22:39:16 (1651531156) [21818.401695] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 22:39:21 (1651531161) [21820.144535] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 22:39:23 (1651531163) [21826.439596] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 22:39:29 (1651531169) [21835.038469] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 22:39:38 (1651531178) [21837.033632] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ========================================================== 22:39:40 (1651531180) [21838.363034] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 22:39:41 (1651531181) [21840.165205] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 22:39:43 (1651531183) [21841.571312] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 22:39:44 (1651531184) [21844.549946] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 22:39:47 (1651531187) [21846.332184] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 22:39:49 (1651531189) [21847.841052] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 22:39:50 (1651531190) [21848.160650] Lustre: DEBUG MARKER: SKIP: sanity test_247f needs >= 2 MDTs [21848.495994] Lustre: DEBUG MARKER: == sanity test 247g: mount striped directory as fileset caches ROOT lookup lock ========================================================== 22:39:51 (1651531191) [21848.816943] Lustre: DEBUG MARKER: SKIP: sanity test_247g needs >= 4 MDTs [21849.156738] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 22:39:52 (1651531192) [21854.100482] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 22:39:57 (1651531197) [21879.812944] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 22:40:22 (1651531222) [21881.152426] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 22:40:24 (1651531224) [21882.487258] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================================== 22:40:25 (1651531225) [21883.843129] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 22:40:26 (1651531226) [21886.125738] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 22:40:29 (1651531229) [22066.436831] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 22:43:29 (1651531409) [22067.717198] Lustre: lustre-OST0001: haven't heard from client f946b48e-ffb4-4e90-99e9-dd076adbf766 (at 10.240.43.64@tcp) in 228 seconds. I think it's dead, and I am evicting it. exp 00000000e85c69dd, cur 1651531411 expire 1651531261 last 1651531183 [22070.769881] Lustre: DEBUG MARKER: == sanity test 255a: check 'lfs ladvise -a willread' ===== 22:43:33 (1651531413) [22071.698391] LustreError: 192753:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 237 sleeping for 4000ms [22075.752878] LustreError: 192753:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 237 awake [22075.766318] LustreError: 192753:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 237 sleeping for 4000ms [22076.080856] LustreError: 192753:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [22274.536049] Lustre: DEBUG MARKER: == sanity test 255b: check 'lfs ladvise -a dontneed' ===== 22:46:57 (1651531617) [22279.227659] Lustre: DEBUG MARKER: == sanity test 255c: suite of ladvise lockahead tests ==== 22:47:02 (1651531622) [22282.363684] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state ========================================================== 22:47:05 (1651531625) [22287.285836] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [22294.005522] Lustre: lustre-OST0001: deleting orphan objects from 0x0:214218 to 0x0:214241 [22294.453093] Lustre: 39100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1651531630/real 1651531630] req@000000006e320d2c x1731745997337600/t0(0) o400->MGC10.240.43.61@tcp@10.240.43.61@tcp:26/25 lens 224/224 e 0 to 1 dl 1651531637 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' [22294.458551] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [22294.464222] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [22294.466102] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b15855451b7bd to 0x7e7b1585546a12c5 [22294.947100] Lustre: DEBUG MARKER: sanity test_256: @@@@@@ FAIL: Changelog catalog has 0 != 2 slots [22300.924449] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 22:47:24 (1651531644) [22315.604093] Lustre: lustre-OST0001: deleting orphan objects from 0x0:214246 to 0x0:214273 [22317.011369] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 22:47:40 (1651531660) [22318.396095] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 22:47:41 (1651531661) [22319.742023] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 22:47:42 (1651531662) [22340.663870] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 22:48:03 (1651531683) [22341.993069] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 22:48:05 (1651531685) [22346.071915] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 22:48:09 (1651531689) [22347.423148] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 22:48:10 (1651531690) [22348.798992] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 22:48:11 (1651531691) [22350.152664] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 22:48:13 (1651531693) [22351.585519] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 22:48:14 (1651531694) [22355.256201] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 22:48:18 (1651531698) [22360.976839] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 22:48:24 (1651531704) [22362.932707] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 22:48:26 (1651531706) [22364.254874] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 22:48:27 (1651531707) [22365.610001] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 22:48:28 (1651531708) [22366.968753] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 22:48:30 (1651531710) [22368.357827] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 22:48:31 (1651531711) [22378.405928] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 22:48:41 (1651531721) [22379.868216] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 22:48:43 (1651531723) [22381.313151] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 22:48:44 (1651531724) [22383.655905] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 22:48:46 (1651531726) [22385.194735] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 22:48:48 (1651531728) [22387.558797] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 22:48:50 (1651531730) [22389.737214] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 22:48:52 (1651531732) [22391.942284] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 22:48:55 (1651531735) [22394.087918] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 22:48:57 (1651531737) [22396.583013] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 22:48:59 (1651531739) [22397.992219] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 22:49:01 (1651531741) [22401.879613] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 22:49:05 (1651531745) [22404.567161] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 22:49:07 (1651531747) [22505.532299] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 22:50:48 (1651531848) [22507.048342] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 22:50:50 (1651531850) [22507.367294] Lustre: DEBUG MARKER: SKIP: sanity test_278 needs >= 2 MDTs [22507.716994] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 22:50:50 (1651531850) [22521.769155] LustreError: 166-1: MGC10.240.43.61@tcp: Connection to MGS (at 10.240.43.61@tcp) was lost; in progress operations using this service will fail [22521.771762] LustreError: Skipped 1 previous similar message [22521.772893] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.240.43.61@tcp) was lost; in progress operations using this service will wait for recovery to complete [22521.775867] Lustre: Skipped 1 previous similar message [22521.776941] Lustre: Evicted from MGS (at 10.240.43.61@tcp) after server handle changed from 0x7e7b1585546a16ed to 0x7e7b1585546d7712 [22521.779164] Lustre: Skipped 1 previous similar message [22521.780863] Lustre: MGC10.240.43.61@tcp: Connection restored to 10.240.43.61@tcp (at 10.240.43.61@tcp) [22521.782590] Lustre: Skipped 3 previous similar messages [22536.093203] Lustre: lustre-OST0001: deleting orphan objects from 0x0:214295 to 0x0:214337 [22548.357207] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 22:51:31 (1651531891) [22548.679422] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [22549.029328] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 22:51:32 (1651531892) [22549.354030] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [22549.704946] Lustre: DEBUG MARKER: == sanity test 300c: chown [22550.051378] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [22550.390320] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 22:51:33 (1651531893) [22550.714769] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [22551.067466] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 22:51:34 (1651531894) [22551.388433] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [22551.738674] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 22:51:34 (1651531894) [22552.045899] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [22552.398608] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 22:51:35 (1651531895) [22552.707822] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [22553.264439] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 22:51:36 (1651531896) [22553.586082] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [22553.929848] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 22:51:37 (1651531897) [22554.248639] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [22554.592922] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 22:51:37 (1651531897) [22554.916741] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [22555.260616] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 22:51:38 (1651531898) [22555.575497] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [22555.929007] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 22:51:39 (1651531899) [22556.248388] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [22556.596026] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 22:51:39 (1651531899) [22558.014301] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 22:51:41 (1651531901) [22558.343198] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [22558.673628] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [22559.025158] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 22:51:42 (1651531902) [22559.341510] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [22559.698359] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 22:51:42 (1651531902) [22560.013086] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [22560.371548] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 22:51:43 (1651531903) [22560.711296] Lustre: DEBUG MARKER: SKIP: sanity test_300r needs >= 2 MDTs [22561.060390] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 22:51:44 (1651531904) [22561.384012] Lustre: DEBUG MARKER: SKIP: sanity test_300s needs >= 2 MDTs [22561.764969] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 22:51:44 (1651531904) [22562.089268] Lustre: DEBUG MARKER: SKIP: sanity test_300t needs at least 2 MDTs [22562.441014] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 22:51:45 (1651531905) [22562.759231] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [22563.109960] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 22:51:46 (1651531906) [22563.424752] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [22563.773470] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 22:51:46 (1651531906) [22564.090230] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [22564.439234] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 22:51:47 (1651531907) [22580.088033] Lustre: DEBUG MARKER: SKIP: sanity test_312 skipping ALWAYS excluded test 312 [22580.437878] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 22:52:03 (1651531923) [22582.230406] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 22:52:05 (1651531925) [22597.042603] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 22:52:20 (1651531940) [22600.583874] Lustre: DEBUG MARKER: == sanity test 316: lfs mv =============================== 22:52:23 (1651531943) [22600.907186] Lustre: DEBUG MARKER: SKIP: sanity test_316 needs >= 2 MDTs [22601.257959] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 22:52:24 (1651531944) [22602.907211] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 22:52:26 (1651531946) [22604.337725] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 22:52:27 (1651531947) [22604.655938] Lustre: DEBUG MARKER: SKIP: sanity test_319 needs >= 2 MDTs [22605.006702] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 22:52:28 (1651531948) [22606.792967] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 22:52:29 (1651531949) [22640.067385] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 22:53:02 (1651531982) [22849.538920] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 22:56:32 (1651532192) [22854.565565] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 22:56:37 (1651532197) [22856.374789] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 22:56:39 (1651532199) [22858.184557] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 22:56:41 (1651532201) [22881.513970] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [22888.888854] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 22:57:12 (1651532232) [22893.321201] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 22:57:16 (1651532236) [22898.204790] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 22:57:21 (1651532241) [22914.321106] Lustre: DEBUG MARKER: SKIP: sanity test_398k 150507472 > 600000 skipping out-of-space test on OST0 [22914.674669] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 22:57:37 (1651532257) [22919.925902] Lustre: DEBUG MARKER: SKIP: sanity test_398l 150489656 > 600000 skipping out-of-space test on OST0 [22920.283400] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 22:57:43 (1651532263) [23039.390230] Lustre: DEBUG MARKER: sanity test_398m: @@@@@@ FAIL: parallel dio write with failure on first stripe succeeded