[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-693.21.1.el7.x86_64 (mockbuild@x86-040.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Fri Feb 23 18:54:16 UTC 2018 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.21.1.el7.x86_64 root=UUID=0020c474-fc12-40a8-8ecc-d5fff4b81556 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fff9fff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffa000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] DMI: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x7fffa max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000080000000 mask 3FFF80000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000f72e0-0x000f72ef] mapped at [ffff8800000f72e0] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x02001000, 0x02001fff] PGTABLE [ 0.000000] BRK [0x02002000, 0x02002fff] PGTABLE [ 0.000000] BRK [0x02003000, 0x02003fff] PGTABLE [ 0.000000] BRK [0x02004000, 0x02004fff] PGTABLE [ 0.000000] BRK [0x02005000, 0x02005fff] PGTABLE [ 0.000000] RAMDISK: [mem 0x356da000-0x36b64fff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f7110 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007ffffb04 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffff177 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffe040 01137 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 000000007fffe000 00040 [ 0.000000] ACPI: SSDT 000000007ffff1eb 00899 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007ffffa84 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fff9fff] [ 0.000000] NODE_DATA(0) allocated [mem 0x7ffd3000-0x7fff9fff] [ 0.000000] Reserving 320MB of memory at 528MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff83001, primary cpu clock [ 0.000000] kvm-clock: using sched offset of 1814932246498 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0x7fff9fff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x7fff9fff] [ 0.000000] On node 0 totalpages: 524184 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 8128 pages used for memmap [ 0.000000] DMA32 zone: 520186 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 35 pages/cpu @ffff88007fc00000 s104600 r8192 d30568 u1048576 [ 0.000000] pcpu-alloc: s104600 r8192 d30568 u1048576 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc13480 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515971 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.21.1.el7.x86_64 root=UUID=0020c474-fc12-40a8-8ecc-d5fff4b81556 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1697716k/2097128k available (6940k kernel code, 392k absent, 399020k reserved, 4560k data, 1792k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.748 MHz processor [ 0.100749] Calibrating delay loop (skipped) preset value.. 5187.49 BogoMIPS (lpj=2593748) [ 0.101711] pid_max: default: 32768 minimum: 301 [ 0.102266] Security Framework initialized [ 0.102749] SELinux: Initializing. [ 0.103178] SELinux: Starting in permissive mode [ 0.103180] Yama: becoming mindful. [ 0.103794] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.105610] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.106886] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes) [ 0.107641] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes) [ 0.108691] Initializing cgroup subsys memory [ 0.109221] Initializing cgroup subsys devices [ 0.109737] Initializing cgroup subsys freezer [ 0.110268] Initializing cgroup subsys net_cls [ 0.110782] Initializing cgroup subsys blkio [ 0.111276] Initializing cgroup subsys perf_event [ 0.111817] Initializing cgroup subsys hugetlb [ 0.112333] Initializing cgroup subsys pids [ 0.112812] Initializing cgroup subsys net_prio [ 0.113422] mce: CPU supports 10 MCE banks [ 0.113942] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.114545] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.115152] tlb_flushall_shift: 6 [ 0.115545] FEATURE SPEC_CTRL Not Present [ 0.116005] FEATURE IBPB_SUPPORT Not Present [ 0.116760] Spectre V2 : Vulnerable: Retpoline without IBPB [ 0.117651] Freeing SMP alternatives: 24k freed [ 0.125187] ACPI: Core revision 20130517 [ 0.126384] ACPI: All ACPI Tables successfully acquired [ 0.127106] ftrace: allocating 26646 entries in 105 pages [ 0.167681] Enabling x2apic [ 0.168043] Enabled x2apic [ 0.168606] Switched APIC routing to physical x2apic. [ 0.170266] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.171060] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.172343] TSC deadline timer enabled [ 0.172355] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.173599] KVM setup paravirtual spinlock [ 0.175616] smpboot: Booting Node 0, Processors #1 OK [ 0.176568] kvm-clock: cpu 1, msr 0:7ff83041, secondary cpu clock [ 0.197490] Brought up 2 CPUs [ 0.197494] KVM setup async PF for cpu 1 [ 0.197498] kvm-stealtime: cpu 1, msr 7fd13480 [ 0.198808] smpboot: Max logical packages: 2 [ 0.199283] smpboot: Total of 2 processors activated (10374.99 BogoMIPS) [ 0.200728] devtmpfs: initialized [ 0.203083] EVM: security.selinux [ 0.203503] EVM: security.ima [ 0.203846] EVM: security.capability [ 0.205487] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.206251] pinctrl core: initialized pinctrl subsystem [ 0.206911] RTC time: 3:51:46, date: 12/14/18 [ 0.207514] NET: Registered protocol family 16 [ 0.208264] ACPI: bus type PCI registered [ 0.208738] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209548] PCI: Using configuration type 1 for base access [ 0.213599] ACPI: Added _OSI(Module Device) [ 0.214085] ACPI: Added _OSI(Processor Device) [ 0.214602] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215117] ACPI: Added _OSI(Processor Aggregator Device) [ 0.216220] ACPI: EC: Look up EC in DSDT [ 0.217543] ACPI: Interpreter enabled [ 0.218001] ACPI: (supports S0 S5) [ 0.218391] ACPI: Using IOAPIC for interrupt routing [ 0.218964] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222370] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.223814] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.224578] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.226045] acpiphp: Slot [3] registered [ 0.226508] acpiphp: Slot [4] registered [ 0.226978] acpiphp: Slot [5] registered [ 0.227460] acpiphp: Slot [6] registered [ 0.227916] acpiphp: Slot [7] registered [ 0.228386] acpiphp: Slot [8] registered [ 0.228845] acpiphp: Slot [9] registered [ 0.229299] acpiphp: Slot [10] registered [ 0.229782] acpiphp: Slot [11] registered [ 0.230263] acpiphp: Slot [12] registered [ 0.230746] acpiphp: Slot [13] registered [ 0.231227] acpiphp: Slot [14] registered [ 0.231699] acpiphp: Slot [15] registered [ 0.232162] acpiphp: Slot [16] registered [ 0.232644] acpiphp: Slot [17] registered [ 0.233111] acpiphp: Slot [18] registered [ 0.233581] acpiphp: Slot [19] registered [ 0.234064] acpiphp: Slot [20] registered [ 0.234550] acpiphp: Slot [21] registered [ 0.235028] acpiphp: Slot [22] registered [ 0.235515] acpiphp: Slot [23] registered [ 0.235981] acpiphp: Slot [24] registered [ 0.236450] acpiphp: Slot [25] registered [ 0.236927] acpiphp: Slot [26] registered [ 0.237399] acpiphp: Slot [27] registered [ 0.237868] acpiphp: Slot [28] registered [ 0.238350] acpiphp: Slot [29] registered [ 0.238820] acpiphp: Slot [30] registered [ 0.239283] acpiphp: Slot [31] registered [ 0.239773] PCI host bridge to bus 0000:00 [ 0.240228] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.240835] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.241562] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.242292] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.243086] pci_bus 0000:00: root bus resource [mem 0x80000000-0xfebfffff window] [ 0.243940] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.244281] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.244760] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.250726] pci 0000:00:01.1: reg 0x20: [io 0xc180-0xc18f] [ 0.253106] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.253898] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.254601] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.255362] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.256209] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300 [ 0.262130] pci 0000:00:01.2: reg 0x20: [io 0xc140-0xc15f] [ 0.264650] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.265005] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.265797] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.266783] pci 0000:00:03.0: [10ec:8139] type 00 class 0x020000 [ 0.269085] pci 0000:00:03.0: reg 0x10: [io 0xc000-0xc0ff] [ 0.271362] pci 0000:00:03.0: reg 0x14: [mem 0xfebc0000-0xfebc00ff] [ 0.282727] pci 0000:00:03.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.282969] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 [ 0.285306] pci 0000:00:04.0: reg 0x10: [io 0xc100-0xc13f] [ 0.287608] pci 0000:00:04.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.299384] pci 0000:00:05.0: [1af4:1002] type 00 class 0x00ff00 [ 0.300593] pci 0000:00:05.0: reg 0x10: [io 0xc160-0xc17f] [ 0.308201] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.309158] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.310094] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.311031] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.311920] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.312930] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.313688] vgaarb: loaded [ 0.314085] SCSI subsystem initialized [ 0.314553] ACPI: bus type USB registered [ 0.315047] usbcore: registered new interface driver usbfs [ 0.315660] usbcore: registered new interface driver hub [ 0.316263] usbcore: registered new device driver usb [ 0.316954] PCI: Using ACPI for IRQ routing [ 0.317438] PCI: pci_cache_line_size set to 64 bytes [ 0.317552] e820: reserve RAM buffer [mem 0x0009f800-0x0009ffff] [ 0.317554] e820: reserve RAM buffer [mem 0x7fffa000-0x7fffffff] [ 0.317661] NetLabel: Initializing [ 0.318056] NetLabel: domain hash size = 128 [ 0.318556] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.319124] NetLabel: unlabeled traffic allowed by default [ 0.319783] amd_nb: Cannot enumerate AMD northbridges [ 0.320358] Switched to clocksource kvm-clock [ 0.328657] pnp: PnP ACPI init [ 0.329052] ACPI: bus type PNP registered [ 0.329592] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.329640] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.329685] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.329725] pnp 00:03: [dma 2] [ 0.329738] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.329830] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.330006] pnp: PnP ACPI: found 5 devices [ 0.330485] ACPI: bus type PNP unregistered [ 0.337968] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.337971] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.337973] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.337976] pci_bus 0000:00: resource 7 [mem 0x80000000-0xfebfffff window] [ 0.338015] NET: Registered protocol family 2 [ 0.338780] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.339673] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.340508] TCP: Hash tables configured (established 16384 bind 16384) [ 0.341289] TCP: reno registered [ 0.341690] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.342378] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.343135] NET: Registered protocol family 1 [ 0.343662] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.344306] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.344976] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.345987] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.347440] PCI: CLS 0 bytes, default 64 [ 0.347501] Unpacking initramfs... [ 0.667062] Freeing initrd memory: 21036k freed [ 0.685515] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.686292] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.687307] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.688027] Initialise system trusted keyring [ 0.688554] audit: initializing netlink socket (disabled) [ 0.689160] type=2000 audit(1544759507.102:1): initialized [ 0.709923] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.710661] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.712453] zpool: loaded [ 0.712782] zbud: loaded [ 0.713318] VFS: Disk quotas dquot_6.5.2 [ 0.713818] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.714702] msgmni has been set to 3356 [ 0.715200] Key type big_key registered [ 0.715663] SELinux: Registering netfilter hooks [ 0.716603] NET: Registered protocol family 38 [ 0.717118] Key type asymmetric registered [ 0.717607] Asymmetric key parser 'x509' registered [ 0.718172] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 0.719037] io scheduler noop registered [ 0.719508] io scheduler deadline registered (default) [ 0.720097] io scheduler cfq registered [ 0.720656] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.721277] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.722075] intel_idle: does not run on family 6 model 42 [ 0.722134] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.722946] ACPI: Power Button [PWRF] [ 0.723501] GHES: HEST is not enabled! [ 0.723997] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.746283] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.747435] Non-volatile memory driver v1.3 [ 0.747930] Linux agpgart interface v0.103 [ 0.748535] crash memory driver: version 1.1 [ 0.749062] rdac: device handler registered [ 0.749617] hp_sw: device handler registered [ 0.750109] emc: device handler registered [ 0.750597] alua: device handler registered [ 0.751107] libphy: Fixed MDIO Bus: probed [ 0.751643] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.752368] ehci-pci: EHCI PCI platform driver [ 0.752894] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.753595] ohci-pci: OHCI PCI platform driver [ 0.754099] uhci_hcd: USB Universal Host Controller Interface driver [ 0.755689] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.756315] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.757149] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.757785] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c140 [ 0.758512] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.759245] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.760150] usb usb1: Product: UHCI Host Controller [ 0.760769] usb usb1: Manufacturer: Linux 3.10.0-693.21.1.el7.x86_64 uhci_hcd [ 0.761572] usb usb1: SerialNumber: 0000:00:01.2 [ 0.762183] hub 1-0:1.0: USB hub found [ 0.762712] hub 1-0:1.0: 2 ports detected [ 0.763310] usbcore: registered new interface driver usbserial [ 0.764027] usbcore: registered new interface driver usbserial_generic [ 0.764806] usbserial: USB Serial support registered for generic [ 0.765510] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.767089] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.767685] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.768443] mousedev: PS/2 mouse device common for all mice [ 0.769417] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.770982] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.772004] rtc_cmos 00:00: RTC can wake from S4 [ 0.772769] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.774002] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.774754] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.775501] cpuidle: using governor menu [ 0.776021] hidraw: raw HID events driver (C) Jiri Kosina [ 0.776733] usbcore: registered new interface driver usbhid [ 0.777368] usbhid: USB HID core driver [ 0.777854] drop_monitor: Initializing network drop monitor service [ 0.778604] TCP: cubic registered [ 0.779005] Initializing XFRM netlink socket [ 0.779588] NET: Registered protocol family 10 [ 0.780338] NET: Registered protocol family 17 [ 0.781031] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.781680] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.782401] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.783496] PM: Hibernation image not present or could not be loaded. [ 0.783500] Loading compiled-in X.509 certificates [ 0.784544] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80' [ 0.786254] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8' [ 0.787871] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: b1c33109fbda94ad2fe1e8e5c1e552bd225760fe' [ 0.789047] registered taskstats version 1 [ 0.792634] Key type trusted registered [ 0.795794] Key type encrypted registered [ 0.798857] IMA: No TPM chip found, activating TPM-bypass! [ 0.799733] Magic number: 14:742:866 [ 0.800279] rtc_cmos 00:00: setting system clock to 2018-12-14 03:51:47 UTC (1544759507) [ 0.802022] Freeing unused kernel memory: 1792k freed [ 0.806590] random: systemd: uninitialized urandom read (16 bytes read) [ 0.808054] random: systemd: uninitialized urandom read (16 bytes read) [ 0.808836] random: systemd: uninitialized urandom read (16 bytes read) [ 0.810359] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.812389] systemd[1]: Detected virtualization kvm. [ 0.812953] systemd[1]: Detected architecture x86-64. [ 0.813584] systemd[1]: Running in initial RAM disk. [ 0.815276] systemd[1]: Set hostname to . [ 0.846102] random: systemd: uninitialized urandom read (16 bytes read) [ 0.846921] random: systemd: uninitialized urandom read (16 bytes read) [ 0.847705] random: systemd: uninitialized urandom read (16 bytes read) [ 0.848499] random: systemd: uninitialized urandom read (16 bytes read) [ 0.849899] random: systemd: uninitialized urandom read (16 bytes read) [ 0.850855] random: systemd: uninitialized urandom read (16 bytes read) [ 0.852026] random: systemd: uninitialized urandom read (16 bytes read) [ 0.857407] systemd[1]: Reached target Swap. [ 0.858718] systemd[1]: Reached target Timers. [ 0.859981] systemd[1]: Reached target Local File Systems. [ 0.861590] systemd[1]: Created slice Root Slice. [ 0.862943] systemd[1]: Listening on Journal Socket. [ 0.864385] systemd[1]: Created slice System Slice. [ 0.866276] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 0.868811] systemd[1]: Starting Journal Service... [ 0.870722] systemd[1]: Starting Apply Kernel Variables... [ 0.872051] systemd[1]: Reached target Slices. [ 0.873323] systemd[1]: Listening on udev Kernel Socket. [ 0.874778] systemd[1]: Listening on udev Control Socket. [ 0.876186] systemd[1]: Reached target Sockets. [ 0.878799] systemd[1]: Starting Setup Virtual Console... [ 0.881019] systemd[1]: Starting dracut cmdline hook... [ 0.882720] systemd[1]: Started Journal Service. [ 1.071570] random: fast init done [ 1.255681] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.261833] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.262554] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.275726] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.276804] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.284506] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000398000, 52:54:00:7a:ee:71, IRQ 11 [ 1.295305] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.303708] libata version 3.00 loaded. [ 1.305676] ata_piix 0000:00:01.1: version 2.13 [ 1.315616] scsi host0: ata_piix [ 1.323627] scsi host1: ata_piix [ 1.324067] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc180 irq 14 [ 1.324839] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc188 irq 15 [ 1.357721] FDC 0 is a S82078B [ 1.553216] virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X [ 1.553238] virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X [ 1.555383] vda: vda1 vda2 [ 1.687394] tsc: Refined TSC clocksource calibration: 2593.627 MHz [ 6.661203] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ 6.890110] systemd-journald[92]: Received SIGTERM from PID 1 (systemd). [ 6.909692] SELinux: Disabled at runtime. [ 6.910237] SELinux: Unregistering netfilter hooks [ 6.951375] type=1404 audit(1544759513.651:2): selinux=0 auid=4294967295 ses=4294967295 [ 6.963875] ip_tables: (C) 2000-2006 Netfilter Core Team [ 6.964636] systemd[1]: Inserted module 'ip_tables' [ 7.061309] EXT4-fs (vda1): re-mounted. Opts: (null) [ 7.076261] systemd-journald[1227]: Received request to flush runtime journal from PID 1 [ 7.692223] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 7.744288] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ 7.854601] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ 7.894418] AES CTR mode by8 optimization enabled [ 7.983499] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.056968] alg: No test for crc32 (crc32-pclmul) [ 8.108591] type=1305 audit(1544759514.808:3): audit_pid=2193 old=0 auid=4294967295 ses=4294967295 res=1 [ 8.145753] ppdev: user-space parallel port driver [ 8.714442] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 8.715578] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 36.719514] random: crng init done [ 395.040368] FS-Cache: Loaded [ 395.061415] RPC: Registered named UNIX socket transport module. [ 395.062114] RPC: Registered udp transport module. [ 395.062646] RPC: Registered tcp transport module. [ 395.063164] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 395.085511] FS-Cache: Netfs 'nfs' registered for caching [ 395.093604] Key type dns_resolver registered [ 395.119692] NFS: Registering the id_resolver key type [ 395.120261] Key type id_resolver registered [ 395.120728] Key type id_legacy registered [ 484.392684] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 484.412780] Loading iSCSI transport class v2.0-870. [ 484.422924] iscsi: registered transport (iser) [ 484.473873] RPC: Registered rdma transport module. [ 484.474477] RPC: Registered rdma backchannel transport module. [ 590.968932] libcfs: loading out-of-tree module taints kernel. [ 590.969876] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 590.973483] LNet: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 [ 590.980366] alg: No test for adler32 (adler32-zlib) [ 590.981006] alg: No test for crc32 (crc32-table) [ 591.741761] Lustre: Lustre: Build Version: 2.11.0 [ 591.774253] LNet: Added LNI 10.9.6.160@tcp [8/256/0/180] [ 591.774944] LNet: Accept secure, port 988 [ 592.037876] Lustre: Mounted lustre-client [ 1817.600484] Lustre: Unmounted lustre-client [ 1950.177082] Lustre: Mounted lustre-client [ 2032.140863] Lustre: 24773:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544761531/real 1544761531] req@ffff88007b6f8300 x1619798364524864/t0(0) o400->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:28/4 lens 224/224 e 0 to 1 dl 1544761538 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2032.143973] Lustre: lustre-OST0000-osc-ffff88007a3bb800: Connection to lustre-OST0000 (at 10.9.6.179@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3357.081616] Lustre: lustre-OST0000-osc-ffff88007a3bb800: Connection restored to 10.9.6.179@tcp (at 10.9.6.179@tcp) [ 3401.275219] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing check_config_client /mnt/lustre [ 3402.318273] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing check_logdir /tmp/test_logs/2018-12-14/044827 [ 3402.852305] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing yml_node [ 3403.348767] Lustre: DEBUG MARKER: Client: Lustre version: 2.11.0 [ 3403.740194] Lustre: DEBUG MARKER: MDS: Lustre version: 2.11.0 [ 3404.134152] Lustre: DEBUG MARKER: OSS: Lustre version: 2.12.0_RC2 [ 3404.374400] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Dec 14 04:48:31 UTC 2018 [ 3404.817370] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing check_config_client /mnt/lustre [ 3406.761730] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 3414.368604] Lustre: DEBUG MARKER: excepting tests: 220 225a 225b 228b 407 253 312 160f 160g 42a 42b 42c [ 3414.603609] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 64b 68 71 115 300o [ 3415.171228] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 04:48:41 (1544762921) [ 3416.349558] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 04:48:42 (1544762922) [ 3417.454984] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 04:48:44 (1544762924) [ 3418.569379] Lustre: DEBUG MARKER: == sanity test 0d: check export proc ================================================================= 04:48:45 (1544762925) [ 3419.945038] Lustre: DEBUG MARKER: sanity test_0d: @@@@@@ FAIL: export client version '134144' != '133888' [ 3428.703883] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ============================================================== 04:48:55 (1544762935) [ 3429.791586] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ==================================================== 04:48:56 (1544762936) [ 3430.890785] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ===================================================== 04:48:57 (1544762937) [ 3431.982222] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ============================== 04:48:58 (1544762938) [ 3433.079770] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 04:48:59 (1544762939) [ 3434.258576] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ========= 04:49:00 (1544762940) [ 3435.346359] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ========= 04:49:01 (1544762941) [ 3436.440924] Lustre: DEBUG MARKER: == sanity test 6e: touch+chgrp ; runas -u 500 -g 500 chgrp (should return error) ==================== 04:49:03 (1544762943) [ 3437.546844] Lustre: DEBUG MARKER: == sanity test 6g: verify new dir in sgid dir inherits group ========================================= 04:49:04 (1544762944) [ 3438.711232] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../ (should return error) =================== 04:49:05 (1544762945) [ 3439.825154] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 04:49:06 (1544762946) [ 3440.939907] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 04:49:07 (1544762947) [ 3442.062733] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 04:49:08 (1544762948) [ 3443.166303] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 04:49:09 (1544762949) [ 3444.268584] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 04:49:10 (1544762950) [ 3445.382965] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 04:49:12 (1544762952) [ 3446.555483] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 04:49:13 (1544762953) [ 3447.674707] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 04:49:14 (1544762954) [ 3448.774973] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 04:49:15 (1544762955) [ 3449.865969] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 04:49:16 (1544762956) [ 3450.962453] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 04:49:17 (1544762957) [ 3452.067577] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 04:49:18 (1544762958) [ 3453.237361] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 04:49:19 (1544762959) [ 3454.349940] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 04:49:20 (1544762960) [ 3455.441233] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 04:49:22 (1544762962) [ 3456.541877] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 04:49:23 (1544762963) [ 3457.647720] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 04:49:24 (1544762964) [ 3458.777779] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 04:49:25 (1544762965) [ 3460.619905] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 04:49:27 (1544762967) [ 3461.905655] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink (should return error) =============================== 04:49:28 (1544762968) [ 3462.108251] LustreError: 26615:0:(symlink.c:94:ll_readlink_internal()) lustre: inode [0x200000401:0x51:0x0]: symlink length 33 not expected 35 [ 3463.189363] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 04:49:29 (1544762969) [ 3464.336607] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ============================= 04:49:30 (1544762970) [ 3465.462639] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 04:49:32 (1544762972) [ 3482.016654] Lustre: lustre-MDT0000-mdc-ffff88007a3bb800: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3489.017498] Lustre: 24773:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544762988/real 1544762988] req@ffff88000950c600 x1619798364626224/t0(0) o400->MGC10.9.6.178@tcp@10.9.6.178@tcp:26/25 lens 224/224 e 0 to 1 dl 1544762995 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3489.020436] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [ 3489.022803] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xe0bed83f37ab91e8 to 0xe0bed83f37ad6392 [ 3489.024398] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 3495.595887] Lustre: lustre-MDT0000-mdc-ffff88007a3bb800: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 3498.721448] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ==================== 04:50:05 (1544763005) [ 3499.317720] Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs [ 3500.408626] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature ============================================== 04:50:07 (1544763007) [ 3520.889263] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3521.022425] Lustre: 24772:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544763020/real 1544763020] req@ffff8800094ced00 x1619798364659680/t0(0) o400->lustre-MDT0000-mdc-ffff88007a3bb800@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544763027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3521.025493] Lustre: lustre-MDT0000-mdc-ffff88007a3bb800: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3521.031065] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [ 3521.049704] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xe0bed83f37ad6392 to 0xe0bed83f37ae2a2b [ 3521.051324] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 3521.153244] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3523.795612] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 04:50:30 (1544763030) [ 3524.935388] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 04:50:31 (1544763031) [ 3526.097958] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 04:50:32 (1544763032) [ 3527.214330] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ========================== 04:50:33 (1544763033) [ 3528.311713] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 04:50:34 (1544763034) [ 3529.409271] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 04:50:36 (1544763036) [ 3530.515532] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 04:50:37 (1544763037) [ 3531.615657] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 04:50:38 (1544763038) [ 3532.774700] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 04:50:39 (1544763039) [ 3533.883445] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 04:50:40 (1544763040) [ 3534.991063] Lustre: DEBUG MARKER: == sanity test 23c: O_APPEND size checks for tiny writes ============================================= 04:50:41 (1544763041) [ 3536.917724] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target =============================================== 04:50:43 (1544763043) [ 3538.036103] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target =================================================== 04:50:44 (1544763044) [ 3539.156289] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================== 04:50:45 (1544763045) [ 3540.293524] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target ============================================== 04:50:46 (1544763046) [ 3541.411950] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 04:50:48 (1544763048) [ 3542.551936] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 04:50:49 (1544763049) [ 3543.678378] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 04:50:50 (1544763050) [ 3544.784262] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 04:50:51 (1544763051) [ 3545.884398] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 04:50:52 (1544763052) [ 3547.015742] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 04:50:53 (1544763053) [ 3548.119381] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 04:50:54 (1544763054) [ 3549.220682] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 04:50:55 (1544763055) [ 3551.132825] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 04:50:57 (1544763057) [ 3552.224488] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 04:50:58 (1544763058) [ 3553.338549] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 04:50:59 (1544763059) [ 3606.933956] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 04:51:53 (1544763113) [ 3608.049594] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 04:51:54 (1544763114) [ 3609.185504] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 04:51:55 (1544763115) [ 3610.299797] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 04:51:56 (1544763116) [ 3611.426058] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 04:51:58 (1544763118) [ 3612.536881] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 04:51:59 (1544763119) [ 3612.562989] Lustre: multiop: using old ioctl(LL_IOC_LOV_GETSTRIPE) on [0x200000401:0x2dc2:0x0], use llapi_layout_get_by_path() [ 3613.701532] Lustre: DEBUG MARKER: == sanity test 24v: list large directory (test hash collision, b=17560) ============================== 04:52:00 (1544763120) [ 3616.105364] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 04:52:02 (1544763122) [ 3617.262028] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ============================================================ 04:52:03 (1544763123) [ 3617.491706] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [ 3618.693687] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ======================================= 04:52:05 (1544763125) [ 3618.938003] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [ 3620.046750] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ===================================== 04:52:06 (1544763126) [ 3634.673858] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================= 04:52:21 (1544763141) [ 3634.905518] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [ 3636.029409] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ========================================================== 04:52:22 (1544763142) [ 3636.262630] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [ 3637.385992] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ============================================================ 04:52:24 (1544763144) [ 3637.617166] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 3638.745987] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 04:52:25 (1544763145) [ 3639.880630] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 04:52:26 (1544763146) [ 3640.979904] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 04:52:27 (1544763147) [ 3642.123716] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 04:52:28 (1544763148) [ 3643.247676] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 04:52:29 (1544763149) [ 3644.383320] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 04:52:31 (1544763151) [ 3645.543288] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 04:52:32 (1544763152) [ 3646.675624] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 04:52:33 (1544763153) [ 3647.842967] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 04:52:34 (1544763154) [ 3648.980556] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file ============================================== 04:52:35 (1544763155) [ 3649.204904] Lustre: DEBUG MARKER: SKIP: sanity test_27b needs >= 2 OSTs [ 3650.305675] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 04:52:36 (1544763156) [ 3651.448047] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 04:52:38 (1544763158) [ 3652.770351] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 04:52:39 (1544763159) [ 3653.890424] Lustre: DEBUG MARKER: == sanity test 27g: /bin/lfs getstripe with no objects =============================================== 04:52:40 (1544763160) [ 3654.995668] Lustre: DEBUG MARKER: == sanity test 27i: /bin/lfs getstripe with some objects ============================================= 04:52:41 (1544763161) [ 3656.104055] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 04:52:42 (1544763162) [ 3657.212559] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 04:52:43 (1544763163) [ 3658.354984] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 04:52:44 (1544763164) [ 3659.470135] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 3659.714422] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 04:52:46 (1544763166) [ 3659.936392] Lustre: DEBUG MARKER: SKIP: sanity test_27n needs >= 2 OSTs [ 3661.045144] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 04:52:47 (1544763167) [ 3661.269762] Lustre: DEBUG MARKER: SKIP: sanity test_27o needs >= 2 OSTs [ 3662.377215] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 04:52:49 (1544763169) [ 3662.598183] Lustre: DEBUG MARKER: SKIP: sanity test_27p needs >= 2 OSTs [ 3663.703790] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 04:52:50 (1544763170) [ 3663.927957] Lustre: DEBUG MARKER: SKIP: sanity test_27q needs >= 2 OSTs [ 3665.042170] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 04:52:51 (1544763171) [ 3665.289251] Lustre: DEBUG MARKER: SKIP: sanity test_27r needs >= 2 OSTs [ 3666.411528] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 04:52:53 (1544763173) [ 3667.524888] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 04:52:54 (1544763174) [ 3668.635487] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================== 04:52:55 (1544763175) [ 3668.858963] Lustre: DEBUG MARKER: SKIP: sanity test_27u needs >= 2 OSTs [ 3670.019658] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 04:52:56 (1544763176) [ 3670.244646] Lustre: DEBUG MARKER: SKIP: sanity test_27v needs >= 2 OSTs [ 3671.366472] Lustre: DEBUG MARKER: == sanity test 27w: check /bin/lfs setstripe -S and getstrip -d options ============================== 04:52:57 (1544763177) [ 3672.503905] Lustre: DEBUG MARKER: == sanity test 27wa: check /bin/lfs setstripe -c -i options ========================================== 04:52:59 (1544763179) [ 3672.737562] Lustre: DEBUG MARKER: SKIP: sanity test_27wa skipping multiple stripe count/offset test [ 3673.862284] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 04:53:00 (1544763180) [ 3674.112371] Lustre: DEBUG MARKER: SKIP: sanity test_27x needs >= 2 OSTs [ 3675.248733] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 04:53:01 (1544763181) [ 3675.471508] Lustre: DEBUG MARKER: SKIP: sanity test_27y needs >= 2 OSTs [ 3676.606291] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 04:53:03 (1544763183) [ 3682.806078] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 3683.069089] Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x417f ver 0x0 [ 3683.293653] Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x417f, count: 1 [ 3683.530261] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:169/0xa9 seq:0 [ 3684.030219] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 3684.285819] Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x4181 ver 0x0 [ 3684.548160] Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x4181, count: 1 [ 3684.788836] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:170/0xaa seq:0 [ 3686.181129] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 04:53:12 (1544763192) [ 3687.565976] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim =============================== 04:53:14 (1544763194) [ 3688.708319] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs ============================================== 04:53:15 (1544763195) [ 3688.931815] Lustre: DEBUG MARKER: SKIP: sanity test_27C needs >= 2 OSTs [ 3690.046199] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API ======================================================== 04:53:16 (1544763196) [ 3690.269871] Lustre: DEBUG MARKER: SKIP: sanity test_27D needs >= 2 OSTs [ 3691.380690] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ==================== 04:53:18 (1544763198) [ 3691.607181] Lustre: DEBUG MARKER: SKIP: sanity test_27E needs >= 2 OSTs [ 3692.726028] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 04:53:19 (1544763199) [ 3692.954341] Lustre: DEBUG MARKER: SKIP: sanity test_27F needs >= 2 OSTs [ 3694.079485] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 04:53:20 (1544763200) [ 3695.323402] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 04:53:21 (1544763201) [ 3698.113417] Lustre: DEBUG MARKER: first d29 [ 3698.354815] Lustre: DEBUG MARKER: second d29 [ 3698.595977] Lustre: DEBUG MARKER: done [ 3699.716201] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 04:53:26 (1544763206) [ 3700.866147] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 04:53:27 (1544763207) [ 3702.065771] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 04:53:28 (1544763208) [ 3703.256221] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 04:53:29 (1544763209) [ 3704.511890] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 04:53:31 (1544763211) [ 3705.665137] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 04:53:32 (1544763212) [ 3706.804268] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 04:53:33 (1544763213) [ 3707.934081] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 04:53:34 (1544763214) [ 3709.061481] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 04:53:35 (1544763215) [ 3716.271648] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 04:53:42 (1544763222) [ 3717.412809] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 04:53:44 (1544763224) [ 3718.559440] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 04:53:45 (1544763225) [ 3719.711858] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 04:53:46 (1544763226) [ 3720.893988] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 04:53:47 (1544763227) [ 3722.058000] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 04:53:48 (1544763228) [ 3723.212700] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ================================================ 04:53:49 (1544763229) [ 3724.368255] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================== 04:53:50 (1544763230) [ 3732.780547] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ================================================= 04:53:59 (1544763239) [ 3733.012348] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [ 3734.273486] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 04:54:00 (1544763240) [ 3734.305836] loop: module loaded [ 3734.309106] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3734.314458] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3736.157554] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 04:54:02 (1544763242) [ 3736.178041] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3736.179095] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3738.410537] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 04:54:05 (1544763245) [ 3738.432841] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3738.434065] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3739.959417] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 04:54:06 (1544763246) [ 3739.982460] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3739.983934] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3747.048163] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 04:54:13 (1544763253) [ 3748.194607] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 04:54:14 (1544763254) [ 3749.354553] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 04:54:15 (1544763255) [ 3750.514186] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 04:54:17 (1544763257) [ 3751.701558] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 04:54:18 (1544763258) [ 3751.722740] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3751.723896] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3757.097623] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 04:54:23 (1544763263) [ 3757.121622] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3757.122728] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3760.112320] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 04:54:26 (1544763266) [ 3760.135230] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3760.136673] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3761.823778] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 04:54:28 (1544763268) [ 3761.846716] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3761.847958] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3763.479986] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 04:54:30 (1544763270) [ 3764.636713] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 04:54:31 (1544763271) [ 3765.805983] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 04:54:32 (1544763272) [ 3766.977333] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 04:54:33 (1544763273) [ 3767.230851] Lustre: DEBUG MARKER: 32p_1 [ 3767.464652] Lustre: DEBUG MARKER: 32p_2 [ 3767.700377] Lustre: DEBUG MARKER: 32p_3 [ 3767.937776] Lustre: DEBUG MARKER: 32p_4 [ 3768.182372] Lustre: DEBUG MARKER: 32p_5 [ 3768.415015] Lustre: DEBUG MARKER: 32p_6 [ 3768.659666] Lustre: DEBUG MARKER: 32p_7 [ 3768.899081] Lustre: DEBUG MARKER: 32p_8 [ 3769.142175] Lustre: DEBUG MARKER: 32p_9 [ 3769.383510] Lustre: DEBUG MARKER: 32p_10 [ 3770.519804] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 04:54:37 (1544763277) [ 3770.547341] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3770.548526] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3773.927644] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 04:54:40 (1544763280) [ 3773.955865] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 3773.957319] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3775.429944] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ================================== 04:54:42 (1544763282) [ 3775.686756] Lustre: DEBUG MARKER: 33_1 [ 3775.925851] Lustre: DEBUG MARKER: 33_2 [ 3777.066692] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 04:54:43 (1544763283) [ 3778.304656] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) =================================== 04:54:44 (1544763284) [ 3779.511449] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 04:54:46 (1544763286) [ 3781.347489] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 04:54:47 (1544763287) [ 3781.580740] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [ 3782.717217] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ================================ 04:54:49 (1544763289) [ 3782.947146] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [ 3784.089435] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 04:54:50 (1544763290) [ 3784.329435] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [ 3785.469107] Lustre: DEBUG MARKER: == sanity test 33g: nonroot user create already existing root created file =========================== 04:54:52 (1544763292) [ 3786.613868] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 04:54:53 (1544763293) [ 3787.799357] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 04:54:54 (1544763294) [ 3788.962828] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 04:54:55 (1544763295) [ 3790.124041] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 04:54:56 (1544763296) [ 3791.283162] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 04:54:57 (1544763297) [ 3792.454196] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 04:54:59 (1544763299) [ 3793.621370] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 04:55:00 (1544763300) [ 3794.791124] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ================================== 04:55:01 (1544763301) [ 3799.099964] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 04:55:05 (1544763305) [ 3800.318266] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 04:55:06 (1544763306) [ 3801.460701] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 04:55:08 (1544763308) [ 3802.616686] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 04:55:09 (1544763309) [ 3803.822874] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 04:55:10 (1544763310) [ 3804.978363] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 04:55:11 (1544763311) [ 3806.117745] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 04:55:12 (1544763312) [ 3808.291740] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 04:55:14 (1544763314) [ 3931.946562] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 04:57:18 (1544763438) [ 3934.122065] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ================================================ 04:57:20 (1544763440) [ 3934.359957] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [ 3935.509521] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 04:57:22 (1544763442) [ 3936.643786] Lustre: DEBUG MARKER: == sanity test 39a: mtime changed on create ========================================================== 04:57:23 (1544763443) [ 3939.798185] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 04:57:26 (1544763446) [ 3942.120273] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 04:57:28 (1544763448) [ 3945.369800] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 04:57:32 (1544763452) [ 3946.534957] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 04:57:33 (1544763453) [ 3947.699957] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 04:57:34 (1544763454) [ 3950.865955] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 04:57:37 (1544763457) [ 3954.027912] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 04:57:40 (1544763460) [ 3956.287421] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 04:57:42 (1544763462) [ 3958.454816] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 04:57:45 (1544763465) [ 3960.155936] LustreError: 13804:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 sleeping for 1000ms [ 3961.156467] LustreError: 13804:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 awake [ 3962.865983] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 04:57:49 (1544763469) [ 3965.174488] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 04:57:51 (1544763471) [ 3973.069192] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 04:57:59 (1544763479) [ 3976.243339] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ================================================== 04:58:02 (1544763482) [ 3987.931685] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ================================= 04:58:14 (1544763494) [ 3989.103930] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========================== 04:58:15 (1544763495) [ 3989.350243] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [ 3990.486178] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime ======================================================= 04:58:17 (1544763497) [ 3991.654182] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 04:58:18 (1544763498) [ 3992.801939] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 04:58:19 (1544763499) [ 3994.935826] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 3995.183530] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 3995.430634] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 3995.685717] Lustre: DEBUG MARKER: == sanity test 42d: test complete truncate of file with cached dirty data ============================ 04:58:22 (1544763502) [ 3998.970728] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 04:58:25 (1544763505) [ 4017.779245] Lustre: DEBUG MARKER: == sanity test 43A: execution of file opened for write should return -ETXTBSY ======================== 04:58:44 (1544763524) [ 4019.980718] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 04:58:46 (1544763526) [ 4021.161144] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 04:58:47 (1544763527) [ 4022.333596] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre ======================================================= 04:58:48 (1544763528) [ 4023.526087] Lustre: DEBUG MARKER: == sanity test 44A: zero length read from a sparse stripe ============================================ 04:58:50 (1544763530) [ 4023.765809] Lustre: DEBUG MARKER: SKIP: sanity test_44A needs >= 2 OSTs [ 4024.945810] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 04:58:51 (1544763531) [ 4026.390376] Lustre: DEBUG MARKER: == sanity test 45: osc io page accounting ============================================================ 04:58:53 (1544763533) [ 4029.384123] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 04:58:56 (1544763536) [ 4032.973636] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 04:58:59 (1544763539) [ 4034.174814] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 04:59:00 (1544763540) [ 4035.348626] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 04:59:01 (1544763541) [ 4036.551304] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 04:59:03 (1544763543) [ 4037.745158] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 04:59:04 (1544763544) [ 4038.943364] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent =================================== 04:59:05 (1544763545) [ 4040.298254] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 04:59:06 (1544763546) [ 4041.467107] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 04:59:08 (1544763548) [ 4042.932844] Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit on create, verify unlink ===================== 04:59:09 (1544763549) [ 4237.761122] Lustre: DEBUG MARKER: == sanity test 51d: check object distribution ======================================================== 05:02:24 (1544763744) [ 4238.003862] Lustre: DEBUG MARKER: SKIP: sanity test_51d needs >= 3 OSTs [ 4239.156008] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =========================================================== 05:02:25 (1544763745) [ 4491.843618] Lustre: DEBUG MARKER: == sanity test 51f: check many open files limit ====================================================== 05:06:38 (1544763998) [ 4877.744826] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 05:13:04 (1544764384) [ 4878.950806] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 05:13:05 (1544764385) [ 4880.143808] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 05:13:06 (1544764386) [ 4880.768397] Lustre: DEBUG MARKER: sanity test_53: @@@@@@ FAIL: can not match last_seq/last_id for *OST*-osc- [ 4893.228313] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 05:13:19 (1544764399) [ 4894.586470] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 05:13:21 (1544764401) [ 4895.787569] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 05:13:22 (1544764402) [ 4896.447685] EXT4-fs (loop3): mounting ext2 file system using the ext4 subsystem [ 4896.457688] EXT4-fs (loop3): mounted filesystem without journal. Opts: (null) [ 4897.786000] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 05:13:24 (1544764404) [ 4898.924059] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 05:13:25 (1544764405) [ 4900.065500] Lustre: DEBUG MARKER: == sanity test 56a: check /bin/lfs getstripe ========================================================= 05:13:26 (1544764406) [ 4900.504985] Lustre: DEBUG MARKER: SKIP: sanity test_56a skip '/bin/lfs getstripe --obd' tests: 1 < 2 [ 4901.652382] Lustre: DEBUG MARKER: == sanity test 56b: check /bin/lfs getdirstripe ====================================================== 05:13:28 (1544764408) [ 4902.834099] Lustre: DEBUG MARKER: == sanity test 56c: check 'lfs df' showing device status ============================================= 05:13:29 (1544764409) [ 4924.706068] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 05:13:51 (1544764431) [ 4925.985992] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 05:13:52 (1544764432) [ 4927.157177] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 05:13:53 (1544764433) [ 4928.336414] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 05:13:55 (1544764435) [ 4929.552343] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 05:13:56 (1544764436) [ 4930.733101] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 05:13:57 (1544764437) [ 4931.907554] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 05:13:58 (1544764438) [ 4933.075957] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 05:13:59 (1544764439) [ 4934.225267] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 05:14:00 (1544764440) [ 4937.618232] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 05:14:04 (1544764444) [ 4938.906803] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 05:14:05 (1544764445) [ 4940.251869] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 05:14:06 (1544764446) [ 4941.959114] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works =============================================== 05:14:08 (1544764448) [ 4942.195325] Lustre: DEBUG MARKER: SKIP: sanity test_56s need at least 2 OSTs [ 4943.342534] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ================================================ 05:14:10 (1544764450) [ 4944.894484] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works =============================================== 05:14:11 (1544764451) [ 4946.256966] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ================================ 05:14:12 (1544764452) [ 4947.670931] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 05:14:14 (1544764454) [ 4947.906050] Lustre: DEBUG MARKER: SKIP: sanity test_56w needs >= 2 OSTs [ 4949.055413] Lustre: DEBUG MARKER: == sanity test 56wb: check lfs_migrate pool support ================================================== 05:14:15 (1544764455) [ 4977.861051] Lustre: DEBUG MARKER: == sanity test 56wc: check unrecognized options for lfs_migrate are passed through =================== 05:14:44 (1544764484) [ 4979.829562] Lustre: DEBUG MARKER: == sanity test 56wd: check lfs_migrate --rsync and --no-rsync work =================================== 05:14:46 (1544764486) [ 4980.066532] Lustre: DEBUG MARKER: SKIP: sanity test_56wd needs >= 2 OSTs [ 4981.218667] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ============================================================ 05:14:47 (1544764487) [ 4981.460086] Lustre: DEBUG MARKER: SKIP: sanity test_56x needs >= 2 OSTs [ 4982.603694] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support =================================================== 05:14:49 (1544764489) [ 4982.844208] Lustre: DEBUG MARKER: SKIP: sanity test_56xa needs >= 2 OSTs [ 4983.997086] Lustre: DEBUG MARKER: == sanity test 56xb: lfs migration hard link support ================================================= 05:14:50 (1544764490) [ 5004.413358] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released ======================================================= 05:15:11 (1544764511) [ 5005.804769] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================== 05:15:12 (1544764512) [ 5007.219067] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir =============================================== 05:15:13 (1544764513) [ 5007.455588] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [ 5008.602489] Lustre: DEBUG MARKER: == sanity test 56ab: lfs find --blocks =============================================================== 05:15:15 (1544764515) [ 5010.495904] Lustre: DEBUG MARKER: == sanity test 56ba: test lfs find --component-end, -start, -count, and -flags ======================= 05:15:17 (1544764517) [ 5012.314803] Lustre: DEBUG MARKER: == sanity test 56ca: check lfs find --mirror-count|-N and --mirror-state ============================= 05:15:19 (1544764519) [ 5015.461531] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 05:15:22 (1544764522) [ 5017.322622] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 05:15:24 (1544764524) [ 5021.833542] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 05:15:28 (1544764528) [ 5022.986407] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 05:15:29 (1544764529) [ 5034.045722] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 05:15:40 (1544764540) [ 5034.450412] Lustre: DEBUG MARKER: test_60 run 26695 - from kernel mode [ 5064.780898] Lustre: DEBUG MARKER: == sanity test 60aa: llog_print works with FIDs and simple names ===================================== 05:16:11 (1544764571) [ 5066.773915] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 05:16:13 (1544764573) [ 5066.783609] LustreError: 11-0: lustre-MDT0000-mdc-ffff88007a3bb800: operation ldlm_enqueue to node 10.9.6.178@tcp failed: rc = -107 [ 5066.784912] Lustre: lustre-MDT0000-mdc-ffff88007a3bb800: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5066.905106] Lustre: lustre-MDT0000-mdc-ffff88007a3bb800: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 5066.906251] Lustre: Skipped 1 previous similar message [ 5068.103106] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 05:16:14 (1544764574) [ 5070.948241] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [ 5070.950650] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xe0bed83f37ae2a2b to 0xe0bed83f38fa427b [ 5070.953365] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 5089.354856] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 05:16:36 (1544764596) [ 5089.366853] Lustre: DEBUG MARKER: test message ID 25992 9885 [ 5090.522508] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================= 05:16:37 (1544764597) [ 5091.857955] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 05:16:38 (1544764598) [ 5093.828065] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 05:16:40 (1544764600) [ 5155.895857] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 05:17:42 (1544764662) [ 5156.628510] Lustre: *** cfs_fail_loc=406, val=0*** [ 5156.629317] LustreError: 31511:0:(osc_request.c:2030:osc_build_rpc()) prep_req failed: -12 [ 5156.630336] LustreError: 31511:0:(osc_cache.c:2286:osc_check_rpcs()) Write request failed with -12 [ 5161.490407] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 05:17:48 (1544764668) [ 5163.243247] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 5163.503007] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ============================================================== 05:17:50 (1544764670) [ 5165.261779] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ========================================================= 05:17:51 (1544764671) [ 5200.619235] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ==================================================== 05:18:27 (1544764707) [ 5201.786889] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 05:18:28 (1544764708) [ 5202.973226] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 0 =================================== 05:18:29 (1544764709) [ 5203.210868] Lustre: DEBUG MARKER: SKIP: sanity test_65c need at least 2 OSTs [ 5204.354234] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 05:18:31 (1544764711) [ 5205.531718] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ===================================================== 05:18:32 (1544764712) [ 5206.705561] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 05:18:33 (1544764713) [ 5207.857766] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 05:18:34 (1544764714) [ 5209.032268] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 05:18:35 (1544764715) [ 5210.205547] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ===================================== 05:18:36 (1544764716) [ 5213.707888] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 05:18:40 (1544764720) [ 5216.227586] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 05:18:42 (1544764722) [ 5216.475478] Lustre: DEBUG MARKER: SKIP: sanity test_65k needs >= 2 OSTs [ 5217.627884] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 05:18:44 (1544764724) [ 5218.800297] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ================================== 05:18:45 (1544764725) [ 5219.954814] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 05:18:46 (1544764726) [ 5222.916628] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 05:18:49 (1544764729) [ 5225.947739] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 5226.202847] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 05:18:52 (1544764732) [ 5227.531178] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 05:18:54 (1544764734) [ 5228.978406] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 05:18:55 (1544764735) [ 5256.211385] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 05:19:22 (1544764762) [ 5257.367467] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 05:19:24 (1544764764) [ 5258.543944] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 05:19:25 (1544764765) [ 5258.555050] Lustre: *** cfs_fail_loc=319, val=0*** [ 5259.699654] Lustre: DEBUG MARKER: == sanity test 76: confirm clients recycle inodes properly =========================================== 05:19:26 (1544764766) [ 5261.731643] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 05:19:28 (1544764768) [ 5263.269239] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ============================================= 05:19:29 (1544764769) [ 5263.385319] Lustre: *** cfs_fail_loc=409, val=0*** [ 5263.579296] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 10.9.6.179@tcp inode [0x2000013a1:0xdcea:0x0] object 0x0:104456 extent [0-4194303], original client csum 1ebe830d (type 4), server csum 1ebe830c (type 4), client csum now 1ebe830c [ 5263.582830] LustreError: 24772:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880007473300 x1619798380400240/t8590040613(8590040613) o4->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544764815 ref 3 fl Interpret:R/4/0 rc 0/0 [ 5264.996393] Lustre: DEBUG MARKER: set checksum type to crc32 [ 5265.127165] Lustre: *** cfs_fail_loc=408, val=0*** [ 5265.131061] LustreError: 133-1: lustre-OST0000-osc-ffff88007a3bb800: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000013a1:0xdcea:0x0] object 0x0:104456 extent [0-4194303], client 5db3c0bc, server 40b2f208, cksum_type 1 [ 5265.133110] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88003e71a100 x1619798380400416/t0(0) o3->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/408 e 0 to 0 dl 1544764817 ref 2 fl Interpret:RM/0/0 rc 4194304/4194304 [ 5266.731181] Lustre: DEBUG MARKER: set checksum type to adler [ 5266.857668] LustreError: 133-1: lustre-OST0000-osc-ffff88007a3bb800: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000013a1:0xdcea:0x0] object 0x0:104456 extent [0-4194303], client e35cd8f5, server cf0ada01, cksum_type 2 [ 5266.859718] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880007473c00 x1619798380400576/t0(0) o3->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/408 e 0 to 0 dl 1544764818 ref 2 fl Interpret:RM/0/0 rc 4194304/4194304 [ 5268.442684] Lustre: DEBUG MARKER: set checksum type to crc32c [ 5268.564654] Lustre: *** cfs_fail_loc=408, val=0*** [ 5268.565240] Lustre: Skipped 1 previous similar message [ 5268.567773] LustreError: 133-1: lustre-OST0000-osc-ffff88007a3bb800: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000013a1:0xdcea:0x0] object 0x0:104456 extent [0-4194303], client bb65054f, server 1ebe830c, cksum_type 4 [ 5270.072032] Lustre: DEBUG MARKER: set checksum type to crc32c [ 5271.264276] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================= 05:19:37 (1544764777) [ 5272.453706] LustreError: 133-1: lustre-OST0000-osc-ffff88007a3bb800: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000013a1:0xdceb:0x0] object 0x0:104457 extent [0-1048575], client 13952d84, server 91c77096, cksum_type 4 [ 5272.455847] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880007470600 x1619798380401328/t0(0) o3->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/408 e 0 to 0 dl 1544764824 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 [ 5272.458525] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 1 previous similar message [ 5279.026927] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================= 05:19:45 (1544764785) [ 5279.118410] Lustre: *** cfs_fail_loc=409, val=0*** [ 5279.118999] Lustre: Skipped 1 previous similar message [ 5279.382665] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 10.9.6.179@tcp inode [0x2000013a1:0xdced:0x0] object 0x0:104458 extent [0-4194303], original client csum de2bf73f (type 4), server csum de2bf73e (type 4), client csum now de2bf73e [ 5279.386058] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88003e71b600 x1619798380401536/t8590040721(8590040721) o4->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544764831 ref 2 fl Interpret:RM/0/0 rc 0/0 [ 5280.816185] LustreError: 133-1: lustre-OST0000-osc-ffff88007a3bb800: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000013a1:0xdced:0x0] object 0x0:104458 extent [0-4194303], client f5a99216, server de2bf73e, cksum_type 4 [ 5283.103017] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 05:19:49 (1544764789) [ 5283.355750] Lustre: DEBUG MARKER: set checksum type to crc32 [ 5283.778658] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000013a1:0xdcef:0x0] object 0x0:104459 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 5287.361085] Lustre: *** cfs_fail_loc=409, val=0*** [ 5287.361676] Lustre: Skipped 11 previous similar messages [ 5287.362262] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000013a1:0xdcef:0x0] object 0x0:104459 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 5287.365237] LustreError: Skipped 3 previous similar messages [ 5290.766730] LustreError: 24772:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880007471800 x1619798380401952/t8590040730(8590040730) o4->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544764842 ref 2 fl Interpret:RM/0/0 rc 0/0 [ 5290.769529] LustreError: 24772:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 7 previous similar messages [ 5295.092997] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000013a1:0xdcef:0x0] object 0x0:104459 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 5295.095998] LustreError: Skipped 3 previous similar messages [ 5307.617035] Lustre: *** cfs_fail_loc=409, val=0*** [ 5307.617627] Lustre: Skipped 15 previous similar messages [ 5307.618199] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000013a1:0xdcef:0x0] object 0x0:104459 extent [4194304-8388607], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 5307.621074] LustreError: Skipped 3 previous similar messages [ 5307.621775] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88003e71aa00 x1619798380402096/t8590040736(8590040736) o4->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544764859 ref 2 fl Interpret:RM/0/0 rc 0/0 [ 5307.624516] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 5 previous similar messages [ 5332.783020] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000013a1:0xdcef:0x0] object 0x0:104459 extent [4194304-8388607], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 5332.785978] LustreError: Skipped 5 previous similar messages [ 5343.142872] Lustre: *** cfs_fail_loc=409, val=0*** [ 5343.143646] Lustre: Skipped 15 previous similar messages [ 5343.144259] LustreError: 24772:0:(osc_request.c:1806:brw_interpret()) lustre-OST0000-osc-ffff88007a3bb800: too many resent retries for object: 0:104459, rc = -11. [ 5343.450284] Lustre: DEBUG MARKER: set checksum type to adler [ 5343.802023] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88003e71b600 x1619798380402384/t8590040746(8590040746) o4->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544764895 ref 2 fl Interpret:RM/0/0 rc 0/0 [ 5343.804830] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 7 previous similar messages [ 5366.944734] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000013a1:0xdcef:0x0] object 0x0:104459 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0 [ 5366.947605] LustreError: Skipped 15 previous similar messages [ 5402.256704] LustreError: 24772:0:(osc_request.c:1806:brw_interpret()) lustre-OST0000-osc-ffff88007a3bb800: too many resent retries for object: 0:104459, rc = -11. [ 5402.258271] LustreError: 24772:0:(osc_request.c:1806:brw_interpret()) Skipped 1 previous similar message [ 5402.571513] Lustre: DEBUG MARKER: set checksum type to crc32c [ 5410.115479] Lustre: *** cfs_fail_loc=409, val=0*** [ 5410.116065] Lustre: Skipped 59 previous similar messages [ 5410.116650] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880007472a00 x1619798380403056/t8590040774(8590040774) o4->lustre-OST0000-osc-ffff88007a3bb800@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544764961 ref 2 fl Interpret:RM/0/0 rc 0/0 [ 5410.119362] LustreError: 24773:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 25 previous similar messages [ 5433.645545] LustreError: 132-0: lustre-OST0000-osc-ffff88007a3bb800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000013a1:0xdcef:0x0] object 0x0:104459 extent [0-4194303], original client csum de2bf73f (type 4), server csum de2bf73e (type 4), client csum now de2bf73f [ 5433.648423] LustreError: Skipped 23 previous similar messages [ 5461.713922] LustreError: 24773:0:(osc_request.c:1806:brw_interpret()) lustre-OST0000-osc-ffff88007a3bb800: too many resent retries for object: 0:104459, rc = -11. [ 5461.715490] LustreError: 24773:0:(osc_request.c:1806:brw_interpret()) Skipped 1 previous similar message [ 5462.018407] Lustre: DEBUG MARKER: set checksum type to crc32c [ 5463.208563] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ================================================ 05:22:49 (1544764969) [ 5465.444767] LustreError: 133-1: lustre-OST0000-osc-ffff88007a3bb800: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000013a1:0xdcf2:0x0] object 0x0:104460 extent [0-4194303], client 1ebe830c, server 537eff9a, cksum_type 4 [ 5468.055475] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 05:22:54 (1544764974) [ 5469.271549] Lustre: Unmounted lustre-client [ 5469.326970] Lustre: *** cfs_fail_loc=40c, val=0*** [ 5469.341483] Lustre: Mounted lustre-client [ 5469.699332] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800648fc800.ost_server_uuid 200 [ 5469.925408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800648fc800.ost_server_uuid in FULL state after 0 sec [ 5470.149055] Lustre: Unmounted lustre-client [ 5470.208355] Lustre: Mounted lustre-client [ 5471.379608] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 05:22:58 (1544764978) [ 5474.082855] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 05:23:00 (1544764980) [ 5486.502319] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 05:23:13 (1544764993) [ 5489.810600] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 05:23:16 (1544764996) [ 5491.360909] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 05:23:18 (1544764998) [ 5492.867747] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ============================================================== 05:23:19 (1544764999) [ 5496.381446] Lustre: DEBUG MARKER: == sanity test 83: Short write in ptask ============================================================== 05:23:23 (1544765003) [ 5497.972243] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ============================================= 05:23:24 (1544765004) [ 5501.263582] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 05:23:28 (1544765008) [ 5502.800512] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads =============================================== 05:23:29 (1544765009) [ 5523.021666] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 05:23:49 (1544765029) [ 5523.258750] Lustre: DEBUG MARKER: SKIP: sanity test_101b needs >= 2 OSTs [ 5524.416354] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ============================================ 05:23:51 (1544765031) [ 5535.438122] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================================== 05:24:02 (1544765042) [ 5666.013631] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ======================= 05:26:12 (1544765172) [ 5673.211071] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ===================================================== 05:26:19 (1544765179) [ 5679.775994] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ==================================================== 05:26:26 (1544765186) [ 5680.800408] Lustre: Unmounted lustre-client [ 5680.859034] Lustre: Mounted lustre-client [ 5692.729516] Lustre: Unmounted lustre-client [ 5692.787408] Lustre: Mounted lustre-client [ 5693.996016] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ================================================================= 05:26:40 (1544765200) [ 5696.303993] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs =========================================== 05:26:43 (1544765203) [ 5696.543139] Lustre: DEBUG MARKER: SKIP: sanity test_102b needs >= 2 OSTs [ 5697.705554] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs =================================== 05:26:44 (1544765204) [ 5697.944068] Lustre: DEBUG MARKER: SKIP: sanity test_102c needs >= 2 OSTs [ 5699.114285] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 05:26:45 (1544765205) [ 5699.354443] Lustre: DEBUG MARKER: SKIP: sanity test_102d needs >= 2 OSTs [ 5700.512475] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts =================================================== 05:26:47 (1544765207) [ 5700.753576] Lustre: DEBUG MARKER: SKIP: sanity test_102f needs >= 2 OSTs [ 5701.913143] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ================================== 05:26:48 (1544765208) [ 5702.178859] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 5702.428166] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 5702.678462] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 5702.930035] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 5704.090121] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ================================= 05:26:50 (1544765210) [ 5704.557439] Lustre: DEBUG MARKER: SKIP: sanity test_102ha large_xattr disabled [ 5705.724326] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ================================================= 05:26:52 (1544765212) [ 5706.890810] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 05:26:53 (1544765213) [ 5707.129628] Lustre: DEBUG MARKER: SKIP: sanity test_102j needs >= 2 OSTs [ 5708.286322] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ===================== 05:26:55 (1544765215) [ 5709.518794] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================= 05:26:56 (1544765216) [ 5710.703654] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 05:26:57 (1544765217) [ 5711.902924] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 05:26:58 (1544765218) [ 5713.685661] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ============================ 05:27:00 (1544765220) [ 5715.077615] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 05:27:01 (1544765221) [ 5716.420950] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values ======================================================= 05:27:03 (1544765223) [ 5717.809225] Lustre: DEBUG MARKER: == sanity test 103a: acl test ======================================================================== 05:27:04 (1544765224) [ 5729.159073] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ==================================================== 05:27:15 (1544765235) [ 5730.467570] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test ======================================================== 05:27:17 (1544765237) [ 5730.505229] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5730.510412] Lustre: lustre-OST0000-osc-ffff880063654800: Connection to lustre-OST0000 (at 10.9.6.179@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5730.514687] LustreError: 167-0: lustre-OST0000-osc-ffff880063654800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.516618] Lustre: lustre-OST0000-osc-ffff880063654800: Connection restored to 10.9.6.179@tcp (at 10.9.6.179@tcp) [ 5730.856319] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880063654800.ost_server_uuid 200 [ 5731.081820] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880063654800.ost_server_uuid in FULL state after 0 sec [ 5732.249141] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 05:27:19 (1544765239) [ 5733.404672] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ======================================== 05:27:20 (1544765240) [ 5734.725490] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ======================================== 05:27:21 (1544765241) [ 5735.916690] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ======================================== 05:27:22 (1544765242) [ 5737.102185] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================== 05:27:23 (1544765243) [ 5737.123624] LustreError: 8361:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 315 sleeping for 10000ms [ 5747.123324] LustreError: 8361:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 315 awake [ 5748.276915] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ======================================== 05:27:35 (1544765255) [ 5749.461696] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 05:27:36 (1544765256) [ 5750.629857] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ================================================================== 05:27:37 (1544765257) [ 5752.821501] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ========================================================= 05:27:39 (1544765259) [ 5754.043354] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 5754.304976] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ================================================== 05:27:41 (1544765261) [ 5754.565663] Lustre: DEBUG MARKER: SKIP: sanity test_116a needs >= 2 OSTs [ 5755.750515] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ===================== 05:27:42 (1544765262) [ 5757.802086] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ================================================================ 05:27:44 (1544765264) [ 5758.982984] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============================================================= 05:27:45 (1544765265) [ 5761.309361] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 05:27:48 (1544765268) [ 5764.234992] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 05:27:50 (1544765270) [ 5765.876202] LustreError: 11-0: lustre-OST0000-osc-ffff880063654800: operation ost_write to node 10.9.6.179@tcp failed: rc = -30 [ 5765.877554] LustreError: 24772:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -30 req@ffff880062462400 x1619798385048816/t0(0) o4->lustre-OST0000-osc-ffff880063654800@10.9.6.179@tcp:6/4 lens 4704/192 e 0 to 0 dl 1544765282 ref 2 fl Interpret:RM/0/0 rc -30/-30 [ 5765.880173] LustreError: 24772:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 15 previous similar messages [ 5768.475927] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 05:27:55 (1544765275) [ 5795.946743] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 05:28:22 (1544765302) [ 5796.276561] Lustre: *** cfs_fail_loc=40a, val=0*** [ 5796.277155] Lustre: Skipped 29 previous similar messages [ 5796.277730] LustreError: 14621:0:(osc_request.c:2030:osc_build_rpc()) prep_req failed: -22 [ 5796.278631] LustreError: 14621:0:(osc_cache.c:2286:osc_check_rpcs()) Write request failed with -22 [ 5798.609143] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ===================================== 05:28:25 (1544765305) [ 5799.044971] LustreError: 15086:0:(osc_request.c:2030:osc_build_rpc()) prep_req failed: -12 [ 5799.045939] LustreError: 15086:0:(osc_cache.c:2286:osc_check_rpcs()) Write request failed with -12 [ 5801.330792] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ================================= 05:28:28 (1544765308) [ 5801.885234] LustreError: 11-0: lustre-OST0000-osc-ffff880063654800: operation ost_write to node 10.9.6.179@tcp failed: rc = -5 [ 5802.886820] LustreError: 11-0: lustre-OST0000-osc-ffff880063654800: operation ost_write to node 10.9.6.179@tcp failed: rc = -5 [ 5804.888712] LustreError: 11-0: lustre-OST0000-osc-ffff880063654800: operation ost_write to node 10.9.6.179@tcp failed: rc = -5 [ 5811.890265] LustreError: 11-0: lustre-OST0000-osc-ffff880063654800: operation ost_write to node 10.9.6.179@tcp failed: rc = -5 [ 5811.891524] LustreError: Skipped 1 previous similar message [ 5811.892115] LustreError: 24772:0:(osc_request.c:1806:brw_interpret()) lustre-OST0000-osc-ffff880063654800: too many resent retries for object: 0:105722, rc = -5. [ 5811.893603] LustreError: 24772:0:(osc_request.c:1806:brw_interpret()) Skipped 1 previous similar message [ 5811.894715] Lustre: 24772:0:(llite_lib.c:2676:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.9.6.178@tcp:/lustre/fid: [0x200001b73:0x4e0:0x0]// may get corrupted (rc -5) [ 5813.859106] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ================================== 05:28:40 (1544765320) [ 5822.466578] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error =========================================== 05:28:49 (1544765329) [ 5823.046154] LustreError: 11-0: lustre-OST0000-osc-ffff880063654800: operation ost_write to node 10.9.6.179@tcp failed: rc = -14 [ 5823.047423] LustreError: Skipped 3 previous similar messages [ 5824.951763] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 05:28:51 (1544765331) [ 5834.477042] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir ======================================================================= 05:29:01 (1544765341) [ 5835.748801] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =================================================================== 05:29:02 (1544765342) [ 5836.996778] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ================================== 05:29:03 (1544765343) [ 5850.619162] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ============================== 05:29:17 (1544765357) [ 5851.911820] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ============================= 05:29:18 (1544765358) [ 5852.155915] Lustre: DEBUG MARKER: SKIP: sanity test_119b needs >= 2 OSTs [ 5853.311966] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ============================================ 05:29:20 (1544765360) [ 5854.642903] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 05:29:21 (1544765361) [ 5856.176436] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 5859.347885] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 05:29:26 (1544765366) [ 5860.901460] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ================================================== 05:29:27 (1544765367) [ 5862.459655] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ==================================================== 05:29:29 (1544765369) [ 5864.023256] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ================================================= 05:29:30 (1544765370) [ 5865.577409] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ================================================== 05:29:32 (1544765372) [ 5874.193041] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ================================================== 05:29:40 (1544765380) [ 5882.787918] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ============================================= 05:29:49 (1544765389) [ 5910.608969] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ================================================================= 05:30:17 (1544765417) [ 5910.636249] Lustre: *** cfs_fail_loc=310, val=0*** [ 5911.806724] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work =========================================================== 05:30:18 (1544765418) [ 5912.400212] Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec [ 5912.766240] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [ 5915.438279] Lustre: DEBUG MARKER: ls 1000 files without statahead: 2 sec [ 5916.248994] Lustre: DEBUG MARKER: ls 1000 files with statahead: 1 sec [ 5940.501562] Lustre: DEBUG MARKER: ls 10000 files without statahead: 13 sec [ 5944.794610] Lustre: DEBUG MARKER: ls 10000 files with statahead: 3 sec [ 5945.049025] Lustre: DEBUG MARKER: ls done [ 5952.961146] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 7 seconds [ 5953.219854] Lustre: DEBUG MARKER: rm done [ 5954.416422] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) =================== 05:31:01 (1544765461) [ 5955.876339] Lustre: *** cfs_fail_loc=803, val=0*** [ 5955.877065] LustreError: 24773:0:(mdc_locks.c:1240:mdc_intent_getattr_async_interpret()) ldlm_cli_enqueue_fini: -110 [ 5957.321813] Lustre: DEBUG MARKER: ls done [ 5959.637671] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ====================================================================== 05:31:06 (1544765466) [ 5959.941062] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 5963.562722] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff880063654800 [ 5963.814931] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff880063654800 [ 5964.058918] Lustre: DEBUG MARKER: LRU=2003 [ 5964.305373] Lustre: DEBUG MARKER: LIMIT=83999 [ 5964.549087] Lustre: DEBUG MARKER: LVF=50323 [ 5964.796771] Lustre: DEBUG MARKER: OLD_LVF=1 [ 5965.042933] Lustre: DEBUG MARKER: Sleep 50 sec [ 6015.332523] Lustre: DEBUG MARKER: Dropped 777 locks in 50s [ 6015.577345] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 6018.615257] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) =================================================== 05:32:05 (1544765525) [ 6024.175908] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 6029.485669] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 6029.729262] Lustre: DEBUG MARKER: lru_size = 200 [ 6040.437563] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 6045.402133] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 6045.649398] Lustre: DEBUG MARKER: lru_size = 4006 [ 6045.892985] Lustre: DEBUG MARKER: lru resize performs the same with no lru resize [ 6050.053454] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ===================================================== 05:32:36 (1544765556) [ 6071.563085] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs =============== 05:32:58 (1544765578) [ 6072.809639] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ================ 05:32:59 (1544765579) [ 6073.991014] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ================================================ 05:33:00 (1544765580) [ 6075.290826] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================== 05:33:02 (1544765582) [ 6076.490753] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================= 05:33:03 (1544765583) [ 6077.716039] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ======================================================== 05:33:04 (1544765584) [ 6119.299959] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ========================================================== 05:33:46 (1544765626) [ 6120.657056] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (1-stripe file) ========================================================== 05:33:47 (1544765627) [ 6120.900374] Lustre: DEBUG MARKER: SKIP: sanity test_130b needs >= 2 OSTs [ 6122.082688] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ================================================ 05:33:48 (1544765628) [ 6122.325120] Lustre: DEBUG MARKER: SKIP: sanity test_130c needs >= 2 OSTs [ 6123.506032] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ========================================================== 05:33:50 (1544765630) [ 6123.746513] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 6124.919780] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 05:33:51 (1544765631) [ 6125.160513] Lustre: DEBUG MARKER: SKIP: sanity test_130e needs >= 2 OSTs [ 6126.353509] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ========================================================= 05:33:53 (1544765633) [ 6127.543935] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ============================ 05:33:54 (1544765634) [ 6128.811860] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ============================================================== 05:33:55 (1544765635) [ 6130.142768] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ============================================= 05:33:56 (1544765636) [ 6131.329611] Lustre: DEBUG MARKER: == sanity test 131d: test short read ================================================================= 05:33:58 (1544765638) [ 6132.568329] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 05:33:59 (1544765639) [ 6133.794539] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ============================================================= 05:34:00 (1544765640) [ 6137.268970] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ======================================================= 05:34:04 (1544765644) [ 6139.727983] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ============================================================= 05:34:06 (1544765646) [ 6161.818845] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================================== 05:34:28 (1544765668) [ 6165.947342] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats ====================================== 05:34:32 (1544765672) [ 6168.983001] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ================= 05:34:35 (1544765675) [ 6169.643558] Lustre: 16188:0:(lprocfs_status.c:2493:lprocfs_wr_root_squash()) lustre: failed to set root_squash due to bad address, rc = -14 [ 6169.645055] Lustre: 16188:0:(lprocfs_status.c:2489:lprocfs_wr_root_squash()) lustre: failed to set root_squash to "", needs uid:gid format, rc = -22 [ 6169.650602] Lustre: 16193:0:(lprocfs_status.c:2561:lprocfs_wr_nosquash_nids()) lustre: failed to set nosquash_nids to "", bad address rc = -14 [ 6169.651985] Lustre: 16193:0:(lprocfs_status.c:2565:lprocfs_wr_nosquash_nids()) lustre: failed to set nosquash_nids due to string too long rc = -22 [ 6169.789913] LustreError: 16244:0:(ldlm_resource.c:365:lru_size_store()) lru_size: invalid value written [ 6169.852402] Lustre: 16319:0:(libcfs_string.c:129:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 6171.034866] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ================= 05:34:37 (1544765677) [ 6197.290996] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6197.531132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6198.696072] LustreError: 11-0: lustre-MDT0000-mdc-ffff880063654800: operation ldlm_enqueue to node 10.9.6.178@tcp failed: rc = -107 [ 6198.697393] LustreError: Skipped 24 previous similar messages [ 6198.698048] Lustre: lustre-MDT0000-mdc-ffff880063654800: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6198.742109] Lustre: lustre-MDT0000-mdc-ffff880063654800: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 6198.961236] Lustre: Unmounted lustre-client [ 6318.622419] LNet: Removed LNI 10.9.6.160@tcp [ 6318.798858] LNet: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 [ 6318.805472] alg: No test for adler32 (adler32-zlib) [ 6318.806070] alg: No test for crc32 (crc32-table) [ 6319.609175] Lustre: Lustre: Build Version: 2.11.0 [ 6319.646481] LNet: Added LNI 10.9.6.160@tcp [8/256/0/180] [ 6319.647232] LNet: Accept secure, port 988 [ 6321.160334] Lustre: 18125:0:(gss_svc_upcall.c:1185:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [ 6321.162025] Lustre: 18125:0:(gss_mech_switch.c:71:lgss_mech_register()) Register gssnull mechanism [ 6321.162968] Key type lgssc registered [ 6321.263643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6353.916696] Lustre: Mounted lustre-client [ 6355.575953] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 6361.954527] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ============================================= 05:37:48 (1544765868) [ 6365.881053] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 05:37:52 (1544765872) [ 6380.591462] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 05:38:07 (1544765887) [ 6486.665518] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 05:39:53 (1544765993) [ 6489.223515] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 05:39:55 (1544765995) [ 6489.517950] Lustre: Unmounted lustre-client [ 6489.579054] Lustre: Mounted lustre-client [ 6490.974610] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls =================================================== 05:39:57 (1544765997) [ 6495.171454] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================== 05:40:01 (1544766001) [ 6496.740411] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================= 05:40:03 (1544766003) [ 6498.107656] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 05:40:04 (1544766004) [ 6499.473251] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ================================================ 05:40:06 (1544766006) [ 6501.106197] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ===================================================================== 05:40:07 (1544766007) [ 6503.147405] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ================================================ 05:40:09 (1544766009) [ 6503.565950] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [ 6504.758015] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 05:40:11 (1544766011) [ 6506.147792] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ============================================================ 05:40:12 (1544766012) [ 6507.740157] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir ============================================== 05:40:14 (1544766014) [ 6509.110797] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea ============================================== 05:40:15 (1544766015) [ 6510.457123] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ========================================================= 05:40:17 (1544766017) [ 6510.725910] brd: module loaded [ 6555.220212] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 05:41:01 (1544766061) [ 6557.589888] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 05:41:04 (1544766064) [ 6559.945409] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 05:41:06 (1544766066) [ 6562.290315] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 05:41:09 (1544766069) [ 6564.620341] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 05:41:11 (1544766071) [ 6574.398315] Lustre: lfs: using old ioctl(LL_IOC_LOV_GETSTRIPE) on [0x200002b12:0x88c:0x0], use llapi_layout_get_by_path() [ 6576.855958] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 05:41:23 (1544766083) [ 6586.416571] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 05:41:33 (1544766093) [ 6596.030340] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 05:41:42 (1544766102) [ 6605.602746] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ========================================================= 05:41:52 (1544766112) [ 6606.843368] Lustre: DEBUG MARKER: Turn on read and write cache [ 6607.532845] Lustre: DEBUG MARKER: Write data and read it back. [ 6607.780854] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 6608.492223] Lustre: DEBUG MARKER: cache hits:: before: 8228, after: 8231 [ 6608.736337] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 6609.230916] Lustre: DEBUG MARKER: cache hits:: before: 8231, after: 8234 [ 6609.477470] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 6610.167553] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 6610.867216] Lustre: DEBUG MARKER: cache hits:: before: 8234, after: 8237 [ 6611.109913] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [ 6611.588794] Lustre: DEBUG MARKER: cache hits:: before: 8237, after: 8237 [ 6611.832212] Lustre: DEBUG MARKER: Write data and read it back. [ 6612.078722] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 6612.790232] Lustre: DEBUG MARKER: cache hits:: before: 8237, after: 8240 [ 6613.034874] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [ 6613.515383] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8240 [ 6613.765015] Lustre: DEBUG MARKER: Turn off read and write cache [ 6614.460247] Lustre: DEBUG MARKER: Write data and read it back [ 6614.713674] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 6615.430803] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8240 [ 6615.685483] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 6616.391186] Lustre: DEBUG MARKER: Write data and read it back [ 6616.647008] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 6617.456430] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8240 [ 6617.713058] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 6618.436026] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8243 [ 6619.956483] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ================================================================ 05:42:06 (1544766126) [ 6620.910277] Lustre: DEBUG MARKER: sanity test_160a: @@@@@@ FAIL: User not found in changelog_users [ 6637.574384] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================= 05:42:24 (1544766144) [ 6638.522887] Lustre: DEBUG MARKER: sanity test_160b: @@@@@@ FAIL: User '' not found in changelog_users [ 6647.509213] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ============================== 05:42:34 (1544766154) [ 6650.182882] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event =============================== 05:42:36 (1544766156) [ 6650.448097] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [ 6651.624168] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) =============================== 05:42:38 (1544766158) [ 6652.400093] Lustre: DEBUG MARKER: sanity test_160e: @@@@@@ FAIL: changelog_deregister failed with 1, expect 2 (ENOENT) [ 6661.204345] Lustre: DEBUG MARKER: SKIP: sanity test_160f skipping ALWAYS excluded test 160f [ 6661.493306] Lustre: DEBUG MARKER: SKIP: sanity test_160g skipping ALWAYS excluded test 160g [ 6661.756000] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ================================================================== 05:42:48 (1544766168) [ 6665.605695] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory =========================================== 05:42:52 (1544766172) [ 6665.843847] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [ 6667.040955] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 05:42:53 (1544766173) [ 6669.493889] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ======================================= 05:42:56 (1544766176) [ 6669.870531] LustreError: 14111:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 140c sleeping for 5000ms [ 6674.871706] LustreError: 14111:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 140c awake [ 6676.545997] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ============================================================== 05:43:03 (1544766183) [ 6677.839313] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ============================================ 05:43:04 (1544766184) [ 6678.082826] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [ 6679.268848] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 05:43:06 (1544766186) [ 6682.854384] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock =================================== 05:43:09 (1544766189) [ 6683.097285] Lustre: DEBUG MARKER: creating a 10 Mb file [ 6683.756344] Lustre: DEBUG MARKER: starting reads [ 6684.129775] Lustre: DEBUG MARKER: truncating the file [ 6684.393082] Lustre: DEBUG MARKER: killing dd [ 6684.635509] Lustre: DEBUG MARKER: removing the temporary file [ 6685.821056] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ============================================= 05:43:12 (1544766192) [ 6685.836338] Lustre: debug daemon will attempt to start writing to /tmp/f170.sanity_log_good (512000kB max) [ 6685.843491] Lustre: shutting down debug daemon thread... [ 6686.841806] Lustre: debug daemon will attempt to start writing to /tmp/f170.sanity_log_good (512000kB max) [ 6686.848254] Lustre: shutting down debug daemon thread... [ 6689.080621] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ============================== 05:43:15 (1544766195) [ 6689.100241] LustreError: 18692:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 50e sleeping for 3000ms [ 6691.086091] Lustre: *** cfs_fail_loc=50e, val=0*** [ 6691.086898] LustreError: dumping log to /tmp/lustre-log.1544766198.22087 [ 6692.101219] LustreError: 18692:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 50e awake [ 6692.102349] LustreError: dumping log to /tmp/lustre-log.1544766199.18692 [ 6693.314965] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ============================================================= 05:43:20 (1544766200) [ 6695.462586] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter ============================================== 05:43:22 (1544766202) [ 6699.283903] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ============================= 05:43:26 (1544766206) [ 6703.889594] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir =========================================================== 05:43:30 (1544766210) [ 6723.752121] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================= 05:43:50 (1544766230) [ 6734.238207] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ========================= 05:44:01 (1544766241) [ 6735.988543] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =============================================================== 05:44:02 (1544766242) [ 6737.715354] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ==================================== 05:44:04 (1544766244) [ 6738.972717] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ================================================ 05:44:05 (1544766245) [ 6744.102931] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap =================================================== 05:44:10 (1544766250) [ 6745.753595] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps =================================== 05:44:12 (1544766252) [ 6747.292825] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ============================================================ 05:44:14 (1544766254) [ 6749.706119] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ======================================================== 05:44:16 (1544766256) [ 6751.783997] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ======================================= 05:44:18 (1544766258) [ 6753.343869] Lustre: DEBUG MARKER: == sanity test 200: OST pools ======================================================================== 05:44:20 (1544766260) [ 6795.975133] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ================================================= 05:45:02 (1544766302) [ 6797.228129] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ============================================ 05:45:04 (1544766304) [ 6798.444140] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset =========================================== 05:45:05 (1544766305) [ 6799.648227] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ============================================= 05:45:06 (1544766306) [ 6800.849884] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ===================================================== 05:45:07 (1544766307) [ 6802.050364] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ================================================ 05:45:08 (1544766308) [ 6803.276667] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset =============================================== 05:45:10 (1544766310) [ 6804.493232] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ================================================= 05:45:11 (1544766311) [ 6805.704048] Lustre: DEBUG MARKER: == sanity test 205: Verify job stats ================================================================= 05:45:12 (1544766312) [ 6815.131407] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity [ 6815.371822] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.15044 [ 6815.959800] Lustre: DEBUG MARKER: sanity test_205: @@@@@@ FAIL: No jobstats for id.205.mkdir.15044 found on mds::*..job_stats [ 6826.964162] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug =============================================== 05:45:33 (1544766333) [ 6828.164256] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse =================================================== 05:45:34 (1544766334) [ 6829.361181] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ====================================================== 05:45:36 (1544766336) [ 6830.631518] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open =================================================================== 05:45:37 (1544766337) [ 6841.850267] Lustre: 22092:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544766341/real 1544766341] req@ffff88003f55f300 x1619804372610992/t0(0) o101->MGC10.9.6.178@tcp@10.9.6.178@tcp:26/25 lens 328/344 e 0 to 1 dl 1544766348 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 6841.853192] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [ 6841.854706] LustreError: 22092:0:(mgc_request.c:603:do_requeue()) failed processing log: -5 [ 6854.656792] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6854.896258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6856.058645] LustreError: 11-0: lustre-MDT0000-mdc-ffff880079063000: operation ldlm_enqueue to node 10.9.6.178@tcp failed: rc = -107 [ 6856.060216] Lustre: lustre-MDT0000-mdc-ffff880079063000: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6856.066594] LustreError: 18110:0:(client.c:2972:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880044139e00 x1619804372610944/t25769881898(25769881898) o101->lustre-MDT0000-mdc-ffff880079063000@10.9.6.178@tcp:12/10 lens 816/544 e 0 to 0 dl 1544766406 ref 2 fl Interpret:RP/4/0 rc 301/301 [ 6856.235470] Lustre: lustre-MDT0000-mdc-ffff880079063000: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 6877.533346] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6877.773136] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6878.936876] LustreError: 11-0: lustre-MDT0000-mdc-ffff880079063000: operation mds_statfs to node 10.9.6.178@tcp failed: rc = -107 [ 6878.938222] Lustre: lustre-MDT0000-mdc-ffff880079063000: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6878.944663] LustreError: 18110:0:(client.c:2972:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88004413aa00 x1619804372611232/t30064771075(30064771075) o101->lustre-MDT0000-mdc-ffff880079063000@10.9.6.178@tcp:12/10 lens 816/544 e 0 to 0 dl 1544766427 ref 2 fl Interpret:RP/4/0 rc 301/301 [ 6878.947509] LustreError: 18110:0:(client.c:2972:ptlrpc_replay_interpret()) Skipped 1 previous similar message [ 6879.011166] Lustre: lustre-MDT0000-mdc-ffff880079063000: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 6880.224659] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly =========================== 05:46:27 (1544766387) [ 6885.743348] bash (2484): drop_caches: 3 [ 6886.249518] Lustre: 18113:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544766348/real 1544766348] req@ffff88004413ad00 x1619804372611024/t0(0) o400->lustre-MDT0000-mdc-ffff880079063000@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544766393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6887.501587] bash (2484): drop_caches: 3 [ 6891.693250] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ==================================================================== 05:46:38 (1544766398) [ 6892.988732] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ====================== 05:46:39 (1544766399) [ 6893.104508] LustreError: 18113:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 40f sleeping for 10000ms [ 6893.107518] LustreError: 18113:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 40f awake [ 6894.291854] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 05:46:41 (1544766401) [ 6897.055958] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ==================== 05:46:43 (1544766403) [ 6897.854129] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xe0bed83f393d798c to 0xe0bed83f395b9b69 [ 6897.855991] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 6898.312087] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ================== 05:46:45 (1544766405) [ 6901.114950] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ================================== 05:46:47 (1544766407) [ 6902.720519] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock =================================== 05:46:49 (1544766409) [ 6902.966260] Lustre: DEBUG MARKER: creating a 10 Mb file [ 6903.620237] Lustre: DEBUG MARKER: starting reads [ 6903.990737] Lustre: DEBUG MARKER: truncating the file [ 6904.244684] Lustre: DEBUG MARKER: killing dd [ 6904.488574] Lustre: DEBUG MARKER: removing the temporary file [ 6905.718948] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ================== 05:46:52 (1544766412) [ 6905.735972] Lustre: *** cfs_fail_loc=411, val=0*** [ 6907.151957] Lustre: DEBUG MARKER: SKIP: sanity test_220 skipping excluded test 220 [ 6907.417145] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM =============================== 05:46:54 (1544766414) [ 6912.088240] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================================= 05:46:58 (1544766418) [ 6913.420545] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ============================== 05:47:00 (1544766420) [ 6914.723639] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ======================================== 05:47:01 (1544766421) [ 6915.786390] Lustre: 18114:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544766379/real 1544766379] req@ffff880044139800 x1619804372611328/t0(0) o400->lustre-MDT0000-mdc-ffff880079063000@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544766422 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6915.997061] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ================================================== 05:47:02 (1544766422) [ 6917.308128] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ================================================== 05:47:04 (1544766424) [ 6918.532707] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ============================== 05:47:05 (1544766425) [ 6957.982885] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a [ 6958.286776] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b [ 6958.556273] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ================================= 05:47:45 (1544766465) [ 6959.822238] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ================ 05:47:46 (1544766466) [ 6960.065158] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [ 6961.254681] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ================================== 05:47:48 (1544766468) [ 6962.560075] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ===================================================== 05:47:49 (1544766469) [ 6962.609207] Lustre: *** cfs_fail_loc=1002, val=0*** [ 7006.030113] Lustre: DEBUG MARKER: SKIP: sanity test_228b skipping excluded test 228b [ 7006.297644] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ================= 05:48:33 (1544766513) [ 7006.354516] Lustre: *** cfs_fail_loc=1002, val=0*** [ 7076.514543] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ============================ 05:49:43 (1544766583) [ 7076.995499] Lustre: DEBUG MARKER: SKIP: sanity test_229 needs >= 2 OSTs [ 7078.190733] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ==================== 05:49:44 (1544766584) [ 7078.432435] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [ 7079.620388] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =============================================================== 05:49:46 (1544766586) [ 7079.863107] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [ 7081.055289] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration is failed ============================= 05:49:47 (1544766587) [ 7081.346464] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [ 7082.539134] Lustre: DEBUG MARKER: == sanity test 230d: check migrate big directory ===================================================== 05:49:49 (1544766589) [ 7082.785970] Lustre: DEBUG MARKER: SKIP: sanity test_230d needs >= 2 MDTs [ 7083.975350] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files =============================================== 05:49:50 (1544766590) [ 7084.227386] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [ 7085.432309] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files ============================================== 05:49:52 (1544766592) [ 7085.689089] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [ 7086.895122] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ==================================================== 05:49:53 (1544766593) [ 7087.149174] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [ 7088.356000] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ============================================================= 05:49:55 (1544766595) [ 7088.599863] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [ 7089.795523] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ======================================= 05:49:56 (1544766596) [ 7090.040398] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [ 7091.246033] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ================ 05:49:58 (1544766598) [ 7094.850325] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ============================ 05:50:01 (1544766601) [ 7099.460628] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ============================================= 05:50:06 (1544766606) [ 7099.649942] LustreError: 11-0: lustre-OST0000-osc-ffff880079063000: operation ldlm_enqueue to node 10.9.6.179@tcp failed: rc = -12 [ 7100.496889] Lustre: Unmounted lustre-client [ 7100.586004] Lustre: Mounted lustre-client [ 7105.343065] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ================================ 05:50:12 (1544766612) [ 7105.551934] LustreError: 11-0: lustre-OST0000-osc-ffff88004ae09000: operation ldlm_enqueue to node 10.9.6.179@tcp failed: rc = -107 [ 7105.553484] Lustre: lustre-OST0000-osc-ffff88004ae09000: Connection to lustre-OST0000 (at 10.9.6.179@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7111.425488] Lustre: lustre-OST0000-osc-ffff88004ae09000: Connection restored to 10.9.6.179@tcp (at 10.9.6.179@tcp) [ 7112.465407] Lustre: Unmounted lustre-client [ 7112.526652] Lustre: Mounted lustre-client [ 7117.307001] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ======================================= 05:50:24 (1544766624) [ 7118.701563] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ==================================== 05:50:25 (1544766625) [ 7120.085010] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM =========================================== 05:50:26 (1544766626) [ 7120.132300] LustreError: 11-0: lustre-OST0000-osc-ffff88004ae0f000: operation ost_setattr to node 10.9.6.179@tcp failed: rc = -107 [ 7120.133594] Lustre: lustre-OST0000-osc-ffff88004ae0f000: Connection to lustre-OST0000 (at 10.9.6.179@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7120.178537] Lustre: lustre-OST0000-osc-ffff88004ae0f000: Connection restored to 10.9.6.179@tcp (at 10.9.6.179@tcp) [ 7120.188532] Lustre: *** cfs_fail_loc=1405, val=0*** [ 7121.397846] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================= 05:50:28 (1544766628) [ 7124.802060] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ================================================ 05:50:31 (1544766631) [ 7126.113398] Lustre: DEBUG MARKER: == sanity test 237: Verify name_to_handle_at/open_by_handle_at syscalls ============================== 05:50:32 (1544766632) [ 7127.318211] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ======================================================== 05:50:34 (1544766634) [ 7128.722440] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test =================================================================== 05:50:35 (1544766635) [ 7146.768508] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ======================================= 05:50:53 (1544766653) [ 7157.476605] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ============================= 05:51:04 (1544766664) [ 7161.592107] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ===================== 05:51:08 (1544766668) [ 7161.834771] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [ 7163.023683] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ====================================================================== 05:51:09 (1544766669) [ 7173.869088] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ====================================================================== 05:51:20 (1544766680) [ 7180.929785] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ======================= 05:51:27 (1544766687) [ 7181.136497] LustreError: 11-0: lustre-MDT0000-mdc-ffff88004ae0f000: operation mds_readpage to node 10.9.6.178@tcp failed: rc = -12 [ 7182.491137] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ========================================================= 05:51:29 (1544766689) [ 7183.009276] Lustre: 6164:0:(file.c:2046:ll_get_grouplock()) group lock already existed with gid 97486 [ 7183.010343] Lustre: 6164:0:(file.c:2115:ll_put_grouplock()) no group lock held [ 7183.011068] Lustre: 6164:0:(file.c:2036:ll_get_grouplock()) group id for group lock must not be 0 [ 7183.012520] Lustre: 6164:0:(file.c:2123:ll_put_grouplock()) group lock 4294967286 doesn't match current id 3543 [ 7199.606142] Lustre: 6164:0:(file.c:2115:ll_put_grouplock()) no group lock held [ 7199.631172] Lustre: 6164:0:(file.c:2036:ll_get_grouplock()) group id for group lock must not be 0 [ 7200.839496] Lustre: DEBUG MARKER: == sanity test 244: sendfile with group lock tests =================================================== 05:51:47 (1544766707) [ 7218.112308] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ============================= 05:52:04 (1544766724) [ 7219.317407] Lustre: DEBUG MARKER: == sanity test 246: Read file of size 4095 should return right length ================================ 05:52:06 (1544766726) [ 7221.072188] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ========================================================= 05:52:07 (1544766727) [ 7221.144055] Lustre: Mounted lustre-client [ 7221.210537] Lustre: Unmounted lustre-client [ 7222.813481] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ================================================ 05:52:09 (1544766729) [ 7222.871042] LustreError: 8575:0:(llite_lib.c:503:client_common_fill_super()) cannot mds_connect: rc = -2 [ 7222.890410] LustreError: 8575:0:(obd_mount.c:1583:lustre_fill_super()) Unable to mount (-2) [ 7224.075928] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside root =================================================== 05:52:10 (1544766730) [ 7224.144624] Lustre: Mounted lustre-client [ 7224.145136] Lustre: Skipped 1 previous similar message [ 7224.209085] Lustre: Unmounted lustre-client [ 7224.209595] Lustre: Skipped 2 previous similar messages [ 7225.671629] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside root ==================================================== 05:52:12 (1544766732) [ 7227.426276] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ============================================================= 05:52:14 (1544766734) [ 7227.586419] LustreError: 11-0: lustre-MDT0000-mdc-ffff880064e64000: operation mds_get_root to node 10.9.6.178@tcp failed: rc = -22 [ 7227.587718] LustreError: 10158:0:(llite_lib.c:503:client_common_fill_super()) cannot mds_connect: rc = -22 [ 7227.605210] LustreError: 10158:0:(obd_mount.c:1583:lustre_fill_super()) Unable to mount (-22) [ 7228.791811] Lustre: DEBUG MARKER: == sanity test 248: fast read verification =========================================================== 05:52:15 (1544766735) [ 7240.221981] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ========================================================= 05:52:27 (1544766747) [ 7241.609343] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ============================================================ 05:52:28 (1544766748) [ 7242.818072] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================== 05:52:29 (1544766749) [ 7242.860230] Lustre: *** cfs_fail_loc=1407, val=0*** [ 7244.086818] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ============================================================= 05:52:30 (1544766750) [ 7246.322797] Lustre: DEBUG MARKER: SKIP: sanity test_253 skipping ALWAYS excluded test 253 [ 7246.591403] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ============================================================= 05:52:33 (1544766753) [ 7247.035098] Lustre: DEBUG MARKER: SKIP: sanity test_254 MDS does not support changelog_size [ 7248.240536] Lustre: DEBUG MARKER: == sanity test 255a: check 'lfs ladvise -a willread' ================================================= 05:52:35 (1544766755) [ 7648.821988] Lustre: DEBUG MARKER: == sanity test 255b: check 'lfs ladvise -a dontneed' ================================================= 05:59:15 (1544767155) [ 7655.986201] Lustre: DEBUG MARKER: == sanity test 255c: suite of ladvise lockahead tests ================================================ 05:59:22 (1544767162) [ 7659.267957] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state =================================== 05:59:26 (1544767166) [ 7662.504369] Lustre: DEBUG MARKER: sanity test_256: @@@@@@ FAIL: Fail to start MDT [ 7673.352948] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ========================================================= 05:59:40 (1544767180) [ 7673.782929] LustreError: 11-0: lustre-MDT0000-mdc-ffff88004ae0f000: operation ldlm_enqueue to node 10.9.6.178@tcp failed: rc = -14 [ 7677.558834] Lustre: lustre-MDT0000-mdc-ffff88004ae0f000: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7684.559559] Lustre: 18113:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544767184/real 1544767184] req@ffff880076639200 x1619804391848080/t0(0) o400->MGC10.9.6.178@tcp@10.9.6.178@tcp:26/25 lens 224/224 e 0 to 1 dl 1544767191 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7684.564157] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [ 7684.570169] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xe0bed83f39880d5d to 0xe0bed83f3995232a [ 7684.575338] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 7684.862984] Lustre: DEBUG MARKER: == sanity test 258a: ================================================================================= 05:59:51 (1544767191) [ 7690.931502] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ================================================ 05:59:57 (1544767197) [ 7692.132612] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ============================================================= 05:59:58 (1544767198) [ 7692.147457] Lustre: *** cfs_fail_loc=806, val=0*** [ 7692.148044] Lustre: 18946:0:(mdc_request.c:856:mdc_close()) lustre-MDT0000-mdc-ffff88004ae0f000: close of FID [0x200003ab4:0x13f6:0x0] failed, file reference will be dropped when this client unmounts or is evicted [ 7692.150004] LustreError: 18946:0:(file.c:205:ll_close_inode_openhandle()) lustre-clilmv-ffff88004ae0f000: inode [0x200003ab4:0x13f6:0x0] mdc close failed: rc = -12 [ 7693.327231] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ================================================== 06:00:00 (1544767200) [ 7698.797925] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ============================= 06:00:05 (1544767205) [ 7700.246710] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests =================================================== 06:00:07 (1544767207) [ 7701.674273] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================== 06:00:08 (1544767208) [ 7703.072129] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test =============================================== 06:00:09 (1544767209) [ 7704.625801] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ============================================= 06:00:11 (1544767211) [ 7707.953652] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ======================================== 06:00:14 (1544767214) [ 7709.353817] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ==================================== 06:00:16 (1544767216) [ 7710.766820] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) =================================== 06:00:17 (1544767217) [ 7712.244273] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================= 06:00:19 (1544767219) [ 7712.483442] Lustre: DEBUG MARKER: SKIP: sanity test_271c skipped until lock convert will be implemented [ 7713.692079] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ================================================ 06:00:20 (1544767220) [ 7714.043532] LustreError: 23736:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 31f sleeping for 4000ms [ 7718.044002] LustreError: 23736:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 31f awake [ 7719.464700] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ================================================ 06:00:26 (1544767226) [ 7728.332771] Lustre: lustre-OST0000-osc-ffff88004ae0f000: Connection to lustre-OST0000 (at 10.9.6.179@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7792.204091] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test =================================================== 06:01:39 (1544767299) [ 7792.634074] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [ 7793.844161] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir =============================================== 06:01:40 (1544767300) [ 7794.270545] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [ 7795.479435] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 7795.908331] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [ 7797.106970] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ==================================== 06:01:43 (1544767303) [ 7797.532176] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [ 7798.725723] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ============================================ 06:01:45 (1544767305) [ 7799.151459] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [ 7800.352117] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ============================================ 06:01:47 (1544767307) [ 7800.784171] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [ 7802.006779] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ============================ 06:01:48 (1544767308) [ 7802.432044] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [ 7803.639625] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory =========================== 06:01:50 (1544767310) [ 7804.080545] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [ 7805.298495] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory =============================== 06:01:52 (1544767312) [ 7805.727241] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [ 7806.932164] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ======================================================== 06:01:53 (1544767313) [ 7807.358603] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [ 7808.601082] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ==================================================== 06:01:55 (1544767315) [ 7809.034217] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [ 7810.236125] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ================= 06:01:57 (1544767317) [ 7810.665380] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [ 7811.873421] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS =========================================== 06:01:58 (1544767318) [ 7813.334887] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA =================== 06:02:00 (1544767320) [ 7813.376060] Lustre: 18110:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544767279/real 1544767279] req@ffff88007aae1e00 x1619804391859632/t0(0) o400->lustre-OST0000-osc-ffff88004ae0f000@10.9.6.179@tcp:28/4 lens 224/224 e 0 to 1 dl 1544767320 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1 [ 7813.458529] Lustre: lustre-OST0000-osc-ffff88004ae0f000: Connection restored to 10.9.6.179@tcp (at 10.9.6.179@tcp) [ 7813.459657] Lustre: Skipped 1 previous similar message [ 7813.766028] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [ 7815.019263] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 7815.289964] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================== 06:02:02 (1544767322) [ 7815.532247] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [ 7816.732684] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ================================== 06:02:03 (1544767323) [ 7816.983887] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [ 7818.196063] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ========================================================= 06:02:05 (1544767325) [ 7818.443100] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [ 7819.646912] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open =============================== 06:02:06 (1544767326) [ 7819.894274] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [ 7821.101731] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ===================================== 06:02:07 (1544767327) [ 7821.344202] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 7822.542893] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ============================ 06:02:09 (1544767329) [ 7822.784547] Lustre: DEBUG MARKER: SKIP: sanity test_311 needs >= 2 OSTs [ 7824.028237] Lustre: DEBUG MARKER: SKIP: sanity test_312 skipping ALWAYS excluded test 312 [ 7824.309421] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ======================================= 06:02:11 (1544767331) [ 7825.890416] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ================================ 06:02:12 (1544767332) [ 7826.133102] Lustre: DEBUG MARKER: SKIP: sanity test_314 needs >= 2 OSTs [ 7827.329974] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ========================================================= 06:02:14 (1544767334) [ 7830.732783] Lustre: DEBUG MARKER: == sanity test 316: lfs mv =========================================================================== 06:02:17 (1544767337) [ 7830.975843] Lustre: DEBUG MARKER: SKIP: sanity test_316 needs >= 2 MDTs [ 7832.182904] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write =============================== 06:02:19 (1544767339) [ 7946.297313] Lustre: DEBUG MARKER: == sanity test 399b: fake read should not be slower than normal read ================================= 06:04:13 (1544767453) [ 7947.928696] Lustre: DEBUG MARKER: == sanity test 400a: Lustre client api program can compile and link ================================== 06:04:14 (1544767454) [ 7951.144131] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ================================================ 06:04:17 (1544767457) [ 7952.684733] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ================== 06:04:19 (1544767459) [ 7953.970237] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ============================== 06:04:20 (1544767460) [ 7955.190571] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. =================== 06:04:22 (1544767462) [ 7956.396120] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' =========================== 06:04:23 (1544767463) [ 7957.618396] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ====================================== 06:04:24 (1544767464) [ 7959.243197] Lustre: DEBUG MARKER: == sanity test 403: i_nlink should not drop to zero due to aliasing ================================== 06:04:26 (1544767466) [ 7959.307662] sysctl (14834): drop_caches: 2 [ 7960.539742] Lustre: DEBUG MARKER: == sanity test 404: validate manual {de}activated works properly for OSPs ============================ 06:04:27 (1544767467) [ 7962.780218] Lustre: DEBUG MARKER: == sanity test 405: Various layout swap lock tests =================================================== 06:04:29 (1544767469) [ 7987.488852] Lustre: 15856:0:(file.c:2046:ll_get_grouplock()) group lock already existed with gid 7356 [ 7988.835732] Lustre: DEBUG MARKER: == sanity test 406: DNE support fs default striping ================================================== 06:04:55 (1544767495) [ 7989.082305] Lustre: DEBUG MARKER: SKIP: sanity test_406 needs >= 2 MDTs [ 7990.332236] Lustre: DEBUG MARKER: SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 7990.610309] Lustre: DEBUG MARKER: == sanity test 408: drop_caches should not hang due to page leaks ==================================== 06:04:57 (1544767497) [ 7990.705673] Lustre: *** cfs_fail_loc=40a, val=0*** [ 7990.706242] LustreError: 18111:0:(osc_request.c:2030:osc_build_rpc()) prep_req failed: -22 [ 7990.707126] LustreError: 18111:0:(osc_cache.c:2310:osc_check_rpcs()) Read request failed with -22 [ 7991.676120] bash (16957): drop_caches: 2 [ 7992.891967] Lustre: DEBUG MARKER: == sanity test 409: Large amount of cross-MDTs hard links on the same file =========================== 06:04:59 (1544767499) [ 7993.140756] Lustre: DEBUG MARKER: SKIP: sanity test_409 We need at least 2 MDTs for this test [ 7994.374147] Lustre: DEBUG MARKER: == sanity test 410: Test inode number returned from kernel thread ==================================== 06:05:01 (1544767501) [ 7994.428536] lustre_kinode_19198: CONFIG_X86_X32 is not set [ 7994.430053] lustre_kinode_19198: inode is 144115440203863280 [ 7994.430885] lustre_kinode_19198: inode is 144115440203863280 [ 7994.431815] lustre_kinode_19198: inode numbers are identical: 144115440203863280 [ 7995.636739] Lustre: DEBUG MARKER: == sanity test 411: Slab allocation error with cgroup does not LBUG ================================== 06:05:02 (1544767502) [ 7999.239146] SLUB: Unable to allocate memory on node -1 (gfp=0x8050) [ 7999.240097] cache: osc_extent_kmem(0:osc_slab_alloc), object size: 176, buffer size: 176, default order: 0, min order: 0 [ 7999.241260] node 0: slabs: 0, objs: 0, free: 0 [ 8000.479684] Lustre: DEBUG MARKER: == sanity test 412: mkdir on specific MDTs =========================================================== 06:05:07 (1544767507) [ 8000.726405] Lustre: DEBUG MARKER: SKIP: sanity test_412 We need at least 2 MDTs for this test [ 8001.933323] Lustre: DEBUG MARKER: == sanity test 413: mkdir on less full MDTs ========================================================== 06:05:08 (1544767508) [ 8002.180104] Lustre: DEBUG MARKER: SKIP: sanity test_413 We need at least 2 MDTs for this test [ 8003.404674] Lustre: DEBUG MARKER: == sanity test 801a: write barrier user interfaces and stat machine ================================== 06:05:10 (1544767510) [ 8004.053199] Lustre: DEBUG MARKER: == sanity test 801b: modification will be blocked by write barrier =================================== 06:05:10 (1544767510) [ 8004.700491] Lustre: DEBUG MARKER: == sanity test 801c: rescan barrier bitmap =========================================================== 06:05:11 (1544767511) [ 8004.945156] Lustre: DEBUG MARKER: SKIP: sanity test_801c needs >= 2 MDTs [ 8006.162500] Lustre: DEBUG MARKER: == sanity test 802: simulate readonly device ========================================================= 06:05:13 (1544767513) [ 8006.810318] Lustre: DEBUG MARKER: == sanity test 803: verify agent object for remote object ============================================ 06:05:13 (1544767513) [ 8007.061782] Lustre: DEBUG MARKER: SKIP: sanity test_803 needs >= 2 MDTs [ 8008.280080] Lustre: DEBUG MARKER: == sanity test 804: verify agent entry for remote entry ============================================== 06:05:15 (1544767515) [ 8008.528258] Lustre: DEBUG MARKER: SKIP: sanity test_804 needs >= 2 MDTs [ 8009.784712] Lustre: DEBUG MARKER: == sanity test 805: ZFS can remove from full fs ====================================================== 06:05:16 (1544767516) [ 8010.054535] Lustre: DEBUG MARKER: SKIP: sanity test_805 ZFS specific test [ 8011.273061] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread =============================== 06:05:18 (1544767518) [ 8011.510476] Lustre: Unmounted lustre-client [ 8011.511056] Lustre: Skipped 5 previous similar messages [ 8027.525444] Key type lgssc unregistered [ 8027.525980] Lustre: 23392:0:(gss_mech_switch.c:80:lgss_mech_unregister()) Unregister krb5 mechanism [ 8029.546484] LNet: Removed LNI 10.9.6.160@tcp [ 8029.851426] LNet: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 [ 8029.857845] alg: No test for adler32 (adler32-zlib) [ 8029.858429] alg: No test for crc32 (crc32-table) [ 8030.765584] Lustre: Lustre: Build Version: 2.11.0 [ 8030.801015] LNet: Added LNI 10.9.6.160@tcp [8/256/0/180] [ 8030.801736] LNet: Accept secure, port 988 [ 8032.358233] Lustre: 23560:0:(gss_svc_upcall.c:1185:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [ 8032.359963] Lustre: 23560:0:(gss_mech_switch.c:71:lgss_mech_register()) Register gssnull mechanism [ 8032.360916] Key type lgssc registered [ 8032.600457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8065.134257] Lustre: Mounted lustre-client [ 8066.813615] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 8073.314742] Lustre: DEBUG MARKER: == sanity test complete, duration 4669 sec =========================================================== 06:06:20 (1544767580) [ 8172.115074] Lustre: 23548:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544767672/real 1544767672] req@ffff88005fb69200 x1619806165272640/t0(0) o400->lustre-MDT0000-mdc-ffff880044219000@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544767679 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 8172.118145] Lustre: lustre-MDT0000-mdc-ffff880044219000: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8172.119923] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [ 9447.059643] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xe0bed83f39982d26 to 0xc586cced8899c0b [ 9447.061092] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 9448.128733] Lustre: lustre-MDT0000-mdc-ffff880044219000: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 9468.782208] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing check_config_client /mnt/lustre [ 9469.797079] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing check_logdir /tmp/test_logs/2018-12-14/062935 [ 9470.324561] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing yml_node [ 9470.805845] Lustre: DEBUG MARKER: Client: Lustre version: 2.11.0 [ 9471.187196] Lustre: DEBUG MARKER: MDS: Lustre version: 2.12.0_RC2 [ 9471.572716] Lustre: DEBUG MARKER: OSS: Lustre version: 2.12.0_RC2 [ 9471.803046] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Dec 14 06:29:38 UTC 2018 [ 9472.248178] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing check_config_client /mnt/lustre [ 9474.339183] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 9475.649037] Lustre: DEBUG MARKER: excepting tests: 220 225a 225b 228b 407 253 312 160f 160g 42a 42b 42c [ 9475.873146] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 64b 68 71 115 300o [ 9476.529944] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 06:29:43 (1544768983) [ 9477.625067] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 06:29:44 (1544768984) [ 9478.715239] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 06:29:45 (1544768985) [ 9479.800580] Lustre: DEBUG MARKER: == sanity test 0d: check export proc ================================================================= 06:29:46 (1544768986) [ 9481.052344] Lustre: DEBUG MARKER: sanity test_0d: @@@@@@ FAIL: export client version '134144' != '133888' [ 9491.877059] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ============================================================== 06:29:58 (1544768998) [ 9492.995828] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ==================================================== 06:29:59 (1544768999) [ 9494.104449] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ===================================================== 06:30:01 (1544769001) [ 9495.224493] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ============================== 06:30:02 (1544769002) [ 9496.327061] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 06:30:03 (1544769003) [ 9497.504387] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ========= 06:30:04 (1544769004) [ 9498.602553] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ========= 06:30:05 (1544769005) [ 9499.759633] Lustre: DEBUG MARKER: == sanity test 6e: touch+chgrp ; runas -u 500 -g 500 chgrp (should return error) ==================== 06:30:06 (1544769006) [ 9500.860872] Lustre: DEBUG MARKER: == sanity test 6g: verify new dir in sgid dir inherits group ========================================= 06:30:07 (1544769007) [ 9502.056132] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../ (should return error) =================== 06:30:09 (1544769009) [ 9503.179275] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 06:30:10 (1544769010) [ 9504.280455] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 06:30:11 (1544769011) [ 9505.380620] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 06:30:12 (1544769012) [ 9506.478559] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 06:30:13 (1544769013) [ 9507.578022] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 06:30:14 (1544769014) [ 9508.679712] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 06:30:15 (1544769015) [ 9509.841664] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 06:30:16 (1544769016) [ 9510.968314] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 06:30:17 (1544769017) [ 9512.088797] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 06:30:19 (1544769019) [ 9513.195740] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 06:30:20 (1544769020) [ 9514.307571] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 06:30:21 (1544769021) [ 9515.403774] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 06:30:22 (1544769022) [ 9516.521449] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 06:30:23 (1544769023) [ 9517.645958] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 06:30:24 (1544769024) [ 9518.754883] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 06:30:25 (1544769025) [ 9519.850345] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 06:30:26 (1544769026) [ 9520.946400] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 06:30:27 (1544769027) [ 9522.066124] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 06:30:29 (1544769029) [ 9523.939430] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 06:30:30 (1544769030) [ 9525.223943] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink (should return error) =============================== 06:30:32 (1544769032) [ 9525.437213] LustreError: 12492:0:(symlink.c:94:ll_readlink_internal()) lustre: inode [0x200005221:0x4f:0x0]: symlink length 33 not expected 35 [ 9526.514678] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 06:30:33 (1544769033) [ 9527.658152] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ============================= 06:30:34 (1544769034) [ 9528.785835] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 06:30:35 (1544769035) [ 9552.499964] LustreError: 11-0: lustre-MDT0000-mdc-ffff880044219000: operation ldlm_enqueue to node 10.9.6.178@tcp failed: rc = -107 [ 9552.501300] Lustre: lustre-MDT0000-mdc-ffff880044219000: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9552.539278] Lustre: lustre-MDT0000-mdc-ffff880044219000: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 9554.050313] Lustre: 23549:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544769054/real 1544769054] req@ffff880009c22400 x1619806165374032/t0(0) o400->lustre-MDT0000-mdc-ffff880044219000@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544769061 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9554.053447] Lustre: 23549:0:(client.c:2100:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9554.054511] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [ 9554.057177] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xc586cced8899c0b to 0xc586cced88bfe3f [ 9555.683360] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ==================== 06:31:02 (1544769062) [ 9556.270722] Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs [ 9557.361535] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature ============================================== 06:31:04 (1544769064) [ 9569.997245] Lustre: 24289:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544769070/real 1544769070] req@ffff88007b5daa00 x1619806165407280/t0(0) o101->MGC10.9.6.178@tcp@10.9.6.178@tcp:26/25 lens 328/344 e 0 to 1 dl 1544769077 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 9570.000138] Lustre: 24289:0:(client.c:2100:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9570.001188] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [ 9577.002902] Lustre: 23548:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544769077/real 1544769077] req@ffff8800433eed00 x1619806165407328/t0(0) o400->lustre-MDT0000-mdc-ffff880044219000@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544769084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9577.006166] Lustre: lustre-MDT0000-mdc-ffff880044219000: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9577.130325] Lustre: lustre-MDT0000-mdc-ffff880044219000: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 9577.131445] Lustre: Skipped 1 previous similar message [ 9577.880846] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9578.130704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9580.732068] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 06:31:27 (1544769087) [ 9581.841963] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 06:31:28 (1544769088) [ 9582.995321] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 06:31:29 (1544769089) [ 9584.112592] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ========================== 06:31:31 (1544769091) [ 9585.211677] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 06:31:32 (1544769092) [ 9586.308349] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 06:31:33 (1544769093) [ 9587.433674] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 06:31:34 (1544769094) [ 9588.544888] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 06:31:35 (1544769095) [ 9589.708586] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 06:31:36 (1544769096) [ 9590.804050] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 06:31:37 (1544769097) [ 9591.911656] Lustre: DEBUG MARKER: == sanity test 23c: O_APPEND size checks for tiny writes ============================================= 06:31:38 (1544769098) [ 9593.841572] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target =============================================== 06:31:40 (1544769100) [ 9594.956378] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target =================================================== 06:31:41 (1544769101) [ 9596.076585] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================== 06:31:43 (1544769103) [ 9597.195688] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target ============================================== 06:31:44 (1544769104) [ 9598.306127] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 06:31:45 (1544769105) [ 9599.430362] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 06:31:46 (1544769106) [ 9600.553098] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 06:31:47 (1544769107) [ 9601.682311] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 06:31:48 (1544769108) [ 9602.007718] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xc586cced88bfe3f to 0xc586cced88c48d7 [ 9602.009145] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [ 9602.800189] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 06:31:49 (1544769109) [ 9603.916259] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 06:31:50 (1544769110) [ 9605.030405] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 06:31:51 (1544769111) [ 9606.153536] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 06:31:53 (1544769113) [ 9607.262014] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 06:31:54 (1544769114) [ 9608.370216] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 06:31:55 (1544769115) [ 9609.488477] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 06:31:56 (1544769116) [ 9662.468585] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 06:32:49 (1544769169) [ 9663.597201] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 06:32:50 (1544769170) [ 9664.737421] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 06:32:51 (1544769171) [ 9665.863709] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 06:32:52 (1544769172) [ 9667.004107] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 06:32:53 (1544769173) [ 9668.127378] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 06:32:55 (1544769175) [ 9668.145547] Lustre: multiop: using old ioctl(LL_IOC_LOV_GETSTRIPE) on [0x200005221:0x2dc0:0x0], use llapi_layout_get_by_path() [ 9669.284837] Lustre: DEBUG MARKER: == sanity test 24v: list large directory (test hash collision, b=17560) ============================== 06:32:56 (1544769176) [ 9671.670304] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 06:32:58 (1544769178) [ 9672.825167] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ============================================================ 06:32:59 (1544769179) [ 9673.069142] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [ 9674.248930] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ======================================= 06:33:01 (1544769181) [ 9674.473426] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [ 9675.586370] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ===================================== 06:33:02 (1544769182) [ 9690.778720] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================= 06:33:17 (1544769197) [ 9691.000763] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [ 9692.112629] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ========================================================== 06:33:19 (1544769199) [ 9692.340735] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [ 9693.460483] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ============================================================ 06:33:20 (1544769200) [ 9693.685098] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 9694.795574] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 06:33:21 (1544769201) [ 9695.925488] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 06:33:22 (1544769202) [ 9697.041036] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 06:33:24 (1544769204) [ 9698.171621] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 06:33:25 (1544769205) [ 9699.301406] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 06:33:26 (1544769206) [ 9700.440795] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 06:33:27 (1544769207) [ 9701.568514] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 06:33:28 (1544769208) [ 9702.695264] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 06:33:29 (1544769209) [ 9703.894852] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 06:33:30 (1544769210) [ 9705.039952] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file ============================================== 06:33:32 (1544769212) [ 9705.268037] Lustre: DEBUG MARKER: SKIP: sanity test_27b needs >= 2 OSTs [ 9706.392252] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 06:33:33 (1544769213) [ 9707.538978] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 06:33:34 (1544769214) [ 9708.861713] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 06:33:35 (1544769215) [ 9710.002795] Lustre: DEBUG MARKER: == sanity test 27g: /bin/lfs getstripe with no objects =============================================== 06:33:36 (1544769216) [ 9711.127326] Lustre: DEBUG MARKER: == sanity test 27i: /bin/lfs getstripe with some objects ============================================= 06:33:38 (1544769218) [ 9712.263391] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 06:33:39 (1544769219) [ 9713.401040] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 06:33:40 (1544769220) [ 9714.548341] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 06:33:41 (1544769221) [ 9715.661201] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 9715.906710] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 06:33:42 (1544769222) [ 9716.133358] Lustre: DEBUG MARKER: SKIP: sanity test_27n needs >= 2 OSTs [ 9717.253658] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 06:33:44 (1544769224) [ 9717.476265] Lustre: DEBUG MARKER: SKIP: sanity test_27o needs >= 2 OSTs [ 9718.593869] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 06:33:45 (1544769225) [ 9718.818127] Lustre: DEBUG MARKER: SKIP: sanity test_27p needs >= 2 OSTs [ 9719.941590] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 06:33:46 (1544769226) [ 9720.167272] Lustre: DEBUG MARKER: SKIP: sanity test_27q needs >= 2 OSTs [ 9721.288806] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 06:33:48 (1544769228) [ 9721.514980] Lustre: DEBUG MARKER: SKIP: sanity test_27r needs >= 2 OSTs [ 9722.657507] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 06:33:49 (1544769229) [ 9723.791263] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 06:33:50 (1544769230) [ 9724.935708] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================== 06:33:51 (1544769231) [ 9725.158973] Lustre: DEBUG MARKER: SKIP: sanity test_27u needs >= 2 OSTs [ 9726.283820] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 06:33:53 (1544769233) [ 9726.512024] Lustre: DEBUG MARKER: SKIP: sanity test_27v needs >= 2 OSTs [ 9727.639073] Lustre: DEBUG MARKER: == sanity test 27w: check /bin/lfs setstripe -S and getstrip -d options ============================== 06:33:54 (1544769234) [ 9728.798337] Lustre: DEBUG MARKER: == sanity test 27wa: check /bin/lfs setstripe -c -i options ========================================== 06:33:55 (1544769235) [ 9729.026364] Lustre: DEBUG MARKER: SKIP: sanity test_27wa skipping multiple stripe count/offset test [ 9730.150619] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 06:33:57 (1544769237) [ 9730.395888] Lustre: DEBUG MARKER: SKIP: sanity test_27x needs >= 2 OSTs [ 9731.523267] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 06:33:58 (1544769238) [ 9731.745564] Lustre: DEBUG MARKER: SKIP: sanity test_27y needs >= 2 OSTs [ 9732.873770] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 06:33:59 (1544769239) [ 9738.909532] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 9739.157577] Lustre: DEBUG MARKER: FID seq 0x200005221, oid 0x417d ver 0x0 [ 9739.381618] Lustre: DEBUG MARKER: LOV seq 0x200005221, oid 0x417d, count: 1 [ 9739.625104] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:193769/0x2f4e9 seq:0 [ 9740.134098] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 9740.386034] Lustre: DEBUG MARKER: FID seq 0x200005221, oid 0x417f ver 0x0 [ 9740.611994] Lustre: DEBUG MARKER: LOV seq 0x200005221, oid 0x417f, count: 1 [ 9740.848990] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:193770/0x2f4ea seq:0 [ 9742.274342] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 06:34:09 (1544769249) [ 9743.625787] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim =============================== 06:34:10 (1544769250) [ 9744.806001] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs ============================================== 06:34:11 (1544769251) [ 9745.033141] Lustre: DEBUG MARKER: SKIP: sanity test_27C needs >= 2 OSTs [ 9746.168542] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API ======================================================== 06:34:13 (1544769253) [ 9746.393870] Lustre: DEBUG MARKER: SKIP: sanity test_27D needs >= 2 OSTs [ 9747.523680] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ==================== 06:34:14 (1544769254) [ 9747.747744] Lustre: DEBUG MARKER: SKIP: sanity test_27E needs >= 2 OSTs [ 9748.886665] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 06:34:15 (1544769255) [ 9749.114553] Lustre: DEBUG MARKER: SKIP: sanity test_27F needs >= 2 OSTs [ 9750.256636] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 06:34:17 (1544769257) [ 9751.522505] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 06:34:18 (1544769258) [ 9753.617512] Lustre: DEBUG MARKER: first d29 [ 9753.858781] Lustre: DEBUG MARKER: second d29 [ 9754.094735] Lustre: DEBUG MARKER: done [ 9755.229444] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 06:34:22 (1544769262) [ 9756.381734] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 06:34:23 (1544769263) [ 9757.547243] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 06:34:24 (1544769264) [ 9758.756045] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 06:34:25 (1544769265) [ 9759.944777] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 06:34:26 (1544769266) [ 9761.095640] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 06:34:28 (1544769268) [ 9762.252352] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 06:34:29 (1544769269) [ 9763.398183] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 06:34:30 (1544769270) [ 9764.551947] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 06:34:31 (1544769271) [ 9771.776052] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 06:34:38 (1544769278) [ 9772.945034] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 06:34:39 (1544769279) [ 9774.125030] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 06:34:41 (1544769281) [ 9775.282961] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 06:34:42 (1544769282) [ 9776.432875] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 06:34:43 (1544769283) [ 9777.594687] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 06:34:44 (1544769284) [ 9778.753644] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ================================================ 06:34:45 (1544769285) [ 9779.903951] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================== 06:34:46 (1544769286) [ 9788.257971] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ================================================= 06:34:55 (1544769295) [ 9788.491145] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [ 9789.666370] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 06:34:56 (1544769296) [ 9789.722995] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9789.724256] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9791.259121] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 06:34:58 (1544769298) [ 9791.285724] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9791.286868] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9792.546917] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 06:34:59 (1544769299) [ 9792.568917] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9792.570333] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9793.896123] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 06:35:00 (1544769300) [ 9793.924586] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9793.926088] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9795.261808] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 06:35:02 (1544769302) [ 9796.427026] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 06:35:03 (1544769303) [ 9797.600730] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 06:35:04 (1544769304) [ 9798.776065] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 06:35:05 (1544769305) [ 9799.953557] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 06:35:06 (1544769306) [ 9799.977308] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9799.978605] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9801.333234] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 06:35:08 (1544769308) [ 9801.360008] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9801.361170] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9802.671061] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 06:35:09 (1544769309) [ 9802.698684] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9802.699909] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9804.047088] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 06:35:11 (1544769311) [ 9804.072978] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9804.074090] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9805.356529] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 06:35:12 (1544769312) [ 9806.534517] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 06:35:13 (1544769313) [ 9807.725102] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 06:35:14 (1544769314) [ 9808.907959] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 06:35:15 (1544769315) [ 9809.142264] Lustre: DEBUG MARKER: 32p_1 [ 9809.378205] Lustre: DEBUG MARKER: 32p_2 [ 9809.616706] Lustre: DEBUG MARKER: 32p_3 [ 9809.859606] Lustre: DEBUG MARKER: 32p_4 [ 9810.104136] Lustre: DEBUG MARKER: 32p_5 [ 9810.338132] Lustre: DEBUG MARKER: 32p_6 [ 9810.576622] Lustre: DEBUG MARKER: 32p_7 [ 9810.814675] Lustre: DEBUG MARKER: 32p_8 [ 9811.058112] Lustre: DEBUG MARKER: 32p_9 [ 9811.298397] Lustre: DEBUG MARKER: 32p_10 [ 9812.448915] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 06:35:19 (1544769319) [ 9812.478782] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9812.480203] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9814.315189] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 06:35:21 (1544769321) [ 9814.346366] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem [ 9814.347527] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9815.874188] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ================================== 06:35:22 (1544769322) [ 9816.136100] Lustre: DEBUG MARKER: 33_1 [ 9816.380795] Lustre: DEBUG MARKER: 33_2 [ 9817.531434] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 06:35:24 (1544769324) [ 9818.754002] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) =================================== 06:35:25 (1544769325) [ 9820.035758] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 06:35:26 (1544769326) [ 9822.008223] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 06:35:28 (1544769328) [ 9822.246025] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [ 9823.399196] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ================================ 06:35:30 (1544769330) [ 9823.632339] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [ 9824.805242] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 06:35:31 (1544769331) [ 9825.045441] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [ 9826.215786] Lustre: DEBUG MARKER: == sanity test 33g: nonroot user create already existing root created file =========================== 06:35:33 (1544769333) [ 9827.394837] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 06:35:34 (1544769334) [ 9828.572863] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 06:35:35 (1544769335) [ 9829.721605] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 06:35:36 (1544769336) [ 9830.885123] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 06:35:37 (1544769337) [ 9832.055037] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 06:35:39 (1544769339) [ 9833.220053] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 06:35:40 (1544769340) [ 9834.400542] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 06:35:41 (1544769341) [ 9835.580995] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ================================== 06:35:42 (1544769342) [ 9839.949817] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 06:35:46 (1544769346) [ 9841.189830] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 06:35:48 (1544769348) [ 9842.342063] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 06:35:49 (1544769349) [ 9843.512321] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 06:35:50 (1544769350) [ 9844.708794] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 06:35:51 (1544769351) [ 9845.873838] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 06:35:52 (1544769352) [ 9847.033577] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 06:35:54 (1544769354) [ 9849.224534] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 06:35:56 (1544769356) [ 9972.892318] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 06:37:59 (1544769479) [ 9975.097755] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ================================================ 06:38:02 (1544769482) [ 9975.336137] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [ 9976.498249] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 06:38:03 (1544769483) [ 9977.649682] Lustre: DEBUG MARKER: == sanity test 39a: mtime changed on create ========================================================== 06:38:04 (1544769484) [ 9980.816616] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 06:38:07 (1544769487) [ 9983.129710] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 06:38:10 (1544769490) [ 9986.330038] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 06:38:13 (1544769493) [ 9987.513979] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 06:38:14 (1544769494) [ 9988.693294] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 06:38:15 (1544769495) [ 9991.872254] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 06:38:18 (1544769498) [ 9995.058615] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 06:38:22 (1544769502) [ 9997.241408] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 06:38:24 (1544769504) [ 9999.467209] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 06:38:26 (1544769506) [10001.168085] LustreError: 32150:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 sleeping for 1000ms [10002.168465] LustreError: 32150:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 awake [10003.869587] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 06:38:30 (1544769510) [10006.249370] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 06:38:33 (1544769513) [10014.149341] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 06:38:41 (1544769521) [10017.337907] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ================================================== 06:38:44 (1544769524) [10029.050260] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ================================= 06:38:56 (1544769536) [10030.228232] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========================== 06:38:57 (1544769537) [10030.463276] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [10031.617736] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime ======================================================= 06:38:58 (1544769538) [10032.778816] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 06:38:59 (1544769539) [10033.930167] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 06:39:00 (1544769540) [10036.076108] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [10036.319017] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [10036.564280] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [10036.818092] Lustre: DEBUG MARKER: == sanity test 42d: test complete truncate of file with cached dirty data ============================ 06:39:03 (1544769543) [10039.844922] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 06:39:06 (1544769546) [10057.327765] Lustre: DEBUG MARKER: == sanity test 43A: execution of file opened for write should return -ETXTBSY ======================== 06:39:24 (1544769564) [10059.521184] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 06:39:26 (1544769566) [10060.795916] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 06:39:27 (1544769567) [10062.026485] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre ======================================================= 06:39:28 (1544769568) [10063.227890] Lustre: DEBUG MARKER: == sanity test 44A: zero length read from a sparse stripe ============================================ 06:39:30 (1544769570) [10063.484767] Lustre: DEBUG MARKER: SKIP: sanity test_44A needs >= 2 OSTs [10064.647268] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 06:39:31 (1544769571) [10066.098300] Lustre: DEBUG MARKER: == sanity test 45: osc io page accounting ============================================================ 06:39:33 (1544769573) [10068.488138] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 06:39:35 (1544769575) [10070.894346] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 06:39:37 (1544769577) [10072.123955] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 06:39:39 (1544769579) [10073.307409] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 06:39:40 (1544769580) [10074.507787] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 06:39:41 (1544769581) [10075.703282] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 06:39:42 (1544769582) [10076.883788] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent =================================== 06:39:43 (1544769583) [10078.253959] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 06:39:45 (1544769585) [10079.415530] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 06:39:46 (1544769586) [10080.884350] Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit on create, verify unlink ===================== 06:39:47 (1544769587) [10275.560421] Lustre: DEBUG MARKER: == sanity test 51d: check object distribution ======================================================== 06:43:02 (1544769782) [10275.797495] Lustre: DEBUG MARKER: SKIP: sanity test_51d needs >= 3 OSTs [10276.959245] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =========================================================== 06:43:03 (1544769783) [10531.708715] Lustre: DEBUG MARKER: == sanity test 51f: check many open files limit ====================================================== 06:47:18 (1544770038) [10917.871414] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 06:53:44 (1544770424) [10919.128134] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 06:53:46 (1544770426) [10920.349224] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 06:53:47 (1544770427) [10920.980408] Lustre: DEBUG MARKER: sanity test_53: @@@@@@ FAIL: can not match last_seq/last_id for *OST*-osc- [10933.192357] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 06:54:00 (1544770440) [10934.412601] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 06:54:01 (1544770441) [10935.573389] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 06:54:02 (1544770442) [10936.281941] EXT4-fs (loop3): mounting ext2 file system using the ext4 subsystem [10936.290228] EXT4-fs (loop3): mounted filesystem without journal. Opts: (null) [10937.569223] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 06:54:04 (1544770444) [10938.729398] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 06:54:05 (1544770445) [10939.886555] Lustre: DEBUG MARKER: == sanity test 56a: check /bin/lfs getstripe ========================================================= 06:54:06 (1544770446) [10940.288599] Lustre: DEBUG MARKER: SKIP: sanity test_56a skip '/bin/lfs getstripe --obd' tests: 1 < 2 [10941.446131] Lustre: DEBUG MARKER: == sanity test 56b: check /bin/lfs getdirstripe ====================================================== 06:54:08 (1544770448) [10942.640308] Lustre: DEBUG MARKER: == sanity test 56c: check 'lfs df' showing device status ============================================= 06:54:09 (1544770449) [10964.516468] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 06:54:31 (1544770471) [10965.796849] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 06:54:32 (1544770472) [10966.982171] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 06:54:34 (1544770474) [10968.154297] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 06:54:35 (1544770475) [10969.390078] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 06:54:36 (1544770476) [10970.557289] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 06:54:37 (1544770477) [10971.724544] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 06:54:38 (1544770478) [10972.888739] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 06:54:39 (1544770479) [10974.101484] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 06:54:41 (1544770481) [10977.130894] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 06:54:44 (1544770484) [10978.433117] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 06:54:45 (1544770485) [10979.742412] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 06:54:46 (1544770486) [10981.585833] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works =============================================== 06:54:48 (1544770488) [10981.820895] Lustre: DEBUG MARKER: SKIP: sanity test_56s need at least 2 OSTs [10982.992788] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ================================================ 06:54:50 (1544770490) [10984.551462] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works =============================================== 06:54:51 (1544770491) [10985.944299] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ================================ 06:54:52 (1544770492) [10987.345783] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 06:54:54 (1544770494) [10987.583470] Lustre: DEBUG MARKER: SKIP: sanity test_56w needs >= 2 OSTs [10988.767709] Lustre: DEBUG MARKER: == sanity test 56wb: check lfs_migrate pool support ================================================== 06:54:55 (1544770495) [11025.459976] Lustre: DEBUG MARKER: == sanity test 56wc: check unrecognized options for lfs_migrate are passed through =================== 06:55:32 (1544770532) [11027.393736] Lustre: DEBUG MARKER: == sanity test 56wd: check lfs_migrate --rsync and --no-rsync work =================================== 06:55:34 (1544770534) [11027.628311] Lustre: DEBUG MARKER: SKIP: sanity test_56wd needs >= 2 OSTs [11028.789441] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ============================================================ 06:55:35 (1544770535) [11029.028821] Lustre: DEBUG MARKER: SKIP: sanity test_56x needs >= 2 OSTs [11030.190260] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support =================================================== 06:55:37 (1544770537) [11030.429198] Lustre: DEBUG MARKER: SKIP: sanity test_56xa needs >= 2 OSTs [11031.594440] Lustre: DEBUG MARKER: == sanity test 56xb: lfs migration hard link support ================================================= 06:55:38 (1544770538) [11051.591212] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released ======================================================= 06:55:58 (1544770558) [11052.982886] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================== 06:56:00 (1544770560) [11054.374763] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir =============================================== 06:56:01 (1544770561) [11054.614712] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [11055.765178] Lustre: DEBUG MARKER: == sanity test 56ab: lfs find --blocks =============================================================== 06:56:02 (1544770562) [11057.733066] Lustre: DEBUG MARKER: == sanity test 56ba: test lfs find --component-end, -start, -count, and -flags ======================= 06:56:04 (1544770564) [11059.587583] Lustre: DEBUG MARKER: == sanity test 56ca: check lfs find --mirror-count|-N and --mirror-state ============================= 06:56:06 (1544770566) [11062.872298] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 06:56:09 (1544770569) [11064.655124] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 06:56:11 (1544770571) [11069.287550] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 06:56:16 (1544770576) [11070.440054] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 06:56:17 (1544770577) [11081.272375] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 06:56:28 (1544770588) [11081.675943] Lustre: DEBUG MARKER: test_60 run 13505 - from kernel mode [11101.921925] Lustre: lustre-MDT0000-mdc-ffff880044219000: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [11116.491060] Lustre: DEBUG MARKER: == sanity test 60aa: llog_print works with FIDs and simple names ===================================== 06:57:03 (1544770623) [11118.516264] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 06:57:05 (1544770625) [11126.920244] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [11126.955705] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xc586cced88c48d7 to 0xc586cced9d521bd [11126.957568] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [11128.244738] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 06:57:15 (1544770635) [11149.915159] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 06:57:36 (1544770656) [11149.926440] Lustre: DEBUG MARKER: test message ID 10983 28331 [11151.116178] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================= 06:57:38 (1544770658) [11152.469950] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 06:57:39 (1544770659) [11154.170689] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 06:57:41 (1544770661) [11216.729641] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 06:58:43 (1544770723) [11217.452485] Lustre: *** cfs_fail_loc=406, val=0*** [11217.453074] LustreError: 17358:0:(osc_request.c:2030:osc_build_rpc()) prep_req failed: -12 [11217.454009] LustreError: 17358:0:(osc_cache.c:2286:osc_check_rpcs()) Write request failed with -12 [11222.047009] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 06:58:49 (1544770729) [11223.868062] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [11224.127954] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ============================================================== 06:58:51 (1544770731) [11226.034315] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ========================================================= 06:58:53 (1544770733) [11270.151963] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ==================================================== 06:59:37 (1544770777) [11271.356114] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 06:59:38 (1544770778) [11272.542133] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 0 =================================== 06:59:39 (1544770779) [11272.783096] Lustre: DEBUG MARKER: SKIP: sanity test_65c need at least 2 OSTs [11273.958639] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 06:59:41 (1544770781) [11275.155647] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ===================================================== 06:59:42 (1544770782) [11276.345004] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 06:59:43 (1544770783) [11277.522073] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 06:59:44 (1544770784) [11278.714526] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 06:59:45 (1544770785) [11279.899667] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ===================================== 06:59:46 (1544770786) [11283.476534] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 06:59:50 (1544770790) [11285.972431] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 06:59:53 (1544770793) [11286.211043] Lustre: DEBUG MARKER: SKIP: sanity test_65k needs >= 2 OSTs [11287.381539] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 06:59:54 (1544770794) [11288.575092] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ================================== 06:59:55 (1544770795) [11289.753117] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 06:59:56 (1544770796) [11293.076179] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 07:00:00 (1544770800) [11296.187712] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [11296.445752] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 07:00:03 (1544770803) [11297.750114] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 07:00:04 (1544770804) [11299.180121] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 07:00:06 (1544770806) [11326.390288] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 07:00:33 (1544770833) [11327.577036] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 07:00:34 (1544770834) [11328.746584] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 07:00:35 (1544770835) [11328.759035] Lustre: *** cfs_fail_loc=319, val=0*** [11329.921019] Lustre: DEBUG MARKER: == sanity test 76: confirm clients recycle inodes properly =========================================== 07:00:36 (1544770836) [11331.969339] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 07:00:39 (1544770839) [11333.551876] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ============================================= 07:00:40 (1544770840) [11333.670890] Lustre: *** cfs_fail_loc=409, val=0*** [11333.771811] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 10.9.6.179@tcp inode [0x2000061c1:0xd568:0x0] object 0x0:296712 extent [0-4194303], original client csum 28a00efb (type 4), server csum 28a00efa (type 4), client csum now 28a00efa [11333.775239] LustreError: 23547:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880064aa7c00 x1619806181041408/t30064880515(30064880515) o4->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544770885 ref 3 fl Interpret:R/4/0 rc 0/0 [11335.170287] Lustre: DEBUG MARKER: set checksum type to crc32 [11335.361174] Lustre: *** cfs_fail_loc=408, val=0*** [11335.364953] LustreError: 133-1: lustre-OST0000-osc-ffff880044219000: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000061c1:0xd568:0x0] object 0x0:296712 extent [0-4194303], client 3a1a57b5, server 8ec50ba3, cksum_type 1 [11335.367010] LustreError: 23547:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880064aa4c00 x1619806181041584/t0(0) o3->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/408 e 0 to 0 dl 1544770887 ref 2 fl Interpret:RM/0/0 rc 4194304/4194304 [11337.071778] Lustre: DEBUG MARKER: set checksum type to adler [11337.267714] LustreError: 133-1: lustre-OST0000-osc-ffff880044219000: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000061c1:0xd568:0x0] object 0x0:296712 extent [0-4194303], client 1e78286e, server a61529a4, cksum_type 2 [11337.269815] LustreError: 23548:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880066751b00 x1619806181041744/t0(0) o3->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/408 e 0 to 0 dl 1544770889 ref 2 fl Interpret:RM/0/0 rc 4194304/4194304 [11338.866256] Lustre: DEBUG MARKER: set checksum type to crc32c [11338.990419] Lustre: *** cfs_fail_loc=408, val=0*** [11338.991006] Lustre: Skipped 1 previous similar message [11338.993469] LustreError: 133-1: lustre-OST0000-osc-ffff880044219000: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000061c1:0xd568:0x0] object 0x0:296712 extent [0-4194303], client 2e2d675b, server 28a00efa, cksum_type 4 [11340.502322] Lustre: DEBUG MARKER: set checksum type to crc32c [11341.708648] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================= 07:00:48 (1544770848) [11342.841306] LustreError: 133-1: lustre-OST0000-osc-ffff880044219000: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000061c1:0xd569:0x0] object 0x0:296713 extent [0-1048575], client 2a4d7c3c, server a5774c30, cksum_type 4 [11342.843415] LustreError: 23545:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880064aa6a00 x1619806181042432/t0(0) o3->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/408 e 0 to 0 dl 1544770894 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 [11342.846079] LustreError: 23545:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 1 previous similar message [11349.549766] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================= 07:00:56 (1544770856) [11349.644275] Lustre: *** cfs_fail_loc=409, val=0*** [11349.644852] Lustre: Skipped 1 previous similar message [11350.009210] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 10.9.6.179@tcp inode [0x2000061c1:0xd56b:0x0] object 0x0:296714 extent [0-4194303], original client csum de2bf73f (type 4), server csum de2bf73e (type 4), client csum now de2bf73e [11350.012618] LustreError: 23547:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880064aa4c00 x1619806181042640/t30064880624(30064880624) o4->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544770901 ref 2 fl Interpret:RM/0/0 rc 0/0 [11351.429842] LustreError: 133-1: lustre-OST0000-osc-ffff880044219000: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000061c1:0xd56b:0x0] object 0x0:296714 extent [0-4194303], client f5a99216, server de2bf73e, cksum_type 4 [11353.749380] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 07:01:00 (1544770860) [11354.001213] Lustre: DEBUG MARKER: set checksum type to crc32 [11354.373013] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000061c1:0xd56d:0x0] object 0x0:296715 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [11358.021659] Lustre: *** cfs_fail_loc=409, val=0*** [11358.022245] Lustre: Skipped 11 previous similar messages [11358.022823] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000061c1:0xd56d:0x0] object 0x0:296715 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [11358.025802] LustreError: Skipped 3 previous similar messages [11358.026405] LustreError: 23548:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880066752a00 x1619806181043040/t30064880630(30064880630) o4->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544770909 ref 2 fl Interpret:RM/0/0 rc 0/0 [11358.029121] LustreError: 23548:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 5 previous similar messages [11365.721367] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000061c1:0xd56d:0x0] object 0x0:296715 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [11365.724240] LustreError: Skipped 3 previous similar messages [11377.170817] Lustre: *** cfs_fail_loc=409, val=0*** [11377.171400] Lustre: Skipped 15 previous similar messages [11377.171976] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000061c1:0xd56d:0x0] object 0x0:296715 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [11377.174794] LustreError: Skipped 3 previous similar messages [11377.175410] LustreError: 23548:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880066752400 x1619806181043168/t30064880638(30064880638) o4->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544770929 ref 2 fl Interpret:RM/0/0 rc 0/0 [11377.178344] LustreError: 23548:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 7 previous similar messages [11402.049670] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000061c1:0xd56d:0x0] object 0x0:296715 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [11402.052541] LustreError: Skipped 5 previous similar messages [11412.336402] Lustre: *** cfs_fail_loc=409, val=0*** [11412.336977] Lustre: Skipped 15 previous similar messages [11412.337533] LustreError: 23548:0:(osc_request.c:1806:brw_interpret()) lustre-OST0000-osc-ffff880044219000: too many resent retries for object: 0:296715, rc = -11. [11412.645406] Lustre: DEBUG MARKER: set checksum type to adler [11413.090379] LustreError: 23545:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880064aa6a00 x1619806181043504/t30064880648(30064880648) o4->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544770965 ref 2 fl Interpret:RM/0/0 rc 0/0 [11413.093220] LustreError: 23545:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 7 previous similar messages [11436.022795] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000061c1:0xd56d:0x0] object 0x0:296715 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0 [11436.025651] LustreError: Skipped 15 previous similar messages [11471.206576] LustreError: 23545:0:(osc_request.c:1806:brw_interpret()) lustre-OST0000-osc-ffff880044219000: too many resent retries for object: 0:296715, rc = -11. [11471.208143] LustreError: 23545:0:(osc_request.c:1806:brw_interpret()) Skipped 1 previous similar message [11471.514738] Lustre: DEBUG MARKER: set checksum type to crc32c [11479.066373] Lustre: *** cfs_fail_loc=409, val=0*** [11479.066946] Lustre: Skipped 59 previous similar messages [11479.067542] LustreError: 23547:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880064aa6100 x1619806181044160/t30064880676(30064880676) o4->lustre-OST0000-osc-ffff880044219000@10.9.6.179@tcp:6/4 lens 608/416 e 0 to 0 dl 1544771030 ref 2 fl Interpret:RM/0/0 rc 0/0 [11479.070375] LustreError: 23547:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 25 previous similar messages [11502.508484] LustreError: 132-0: lustre-OST0000-osc-ffff880044219000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.9.6.179@tcp inode [0x2000061c1:0xd56d:0x0] object 0x0:296715 extent [4194304-8388607], original client csum de2bf73f (type 4), server csum de2bf73e (type 4), client csum now de2bf73f [11502.511446] LustreError: Skipped 23 previous similar messages [11530.540862] LustreError: 23547:0:(osc_request.c:1806:brw_interpret()) lustre-OST0000-osc-ffff880044219000: too many resent retries for object: 0:296715, rc = -11. [11530.542439] LustreError: 23547:0:(osc_request.c:1806:brw_interpret()) Skipped 1 previous similar message [11530.843960] Lustre: DEBUG MARKER: set checksum type to crc32c [11532.026703] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ================================================ 07:03:59 (1544771039) [11534.291682] LustreError: 133-1: lustre-OST0000-osc-ffff880044219000: BAD READ CHECKSUM: from 10.9.6.179@tcp inode [0x2000061c1:0xd56f:0x0] object 0x0:296716 extent [0-4194303], client 28a00efa, server 7b340e5d, cksum_type 4 [11536.918110] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 07:04:03 (1544771043) [11538.022715] Lustre: Unmounted lustre-client [11538.078460] Lustre: *** cfs_fail_loc=40c, val=0*** [11538.090127] Lustre: Mounted lustre-client [11538.445015] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88005fb5b800.ost_server_uuid 200 [11538.675959] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88005fb5b800.ost_server_uuid in FULL state after 0 sec [11538.901019] Lustre: Unmounted lustre-client [11538.960630] Lustre: Mounted lustre-client [11540.147337] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 07:04:07 (1544771047) [11542.882502] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 07:04:09 (1544771049) [11555.236379] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 07:04:22 (1544771062) [11558.290377] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 07:04:25 (1544771065) [11559.777229] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 07:04:26 (1544771066) [11561.261107] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ============================================================== 07:04:28 (1544771068) [11564.817562] Lustre: DEBUG MARKER: == sanity test 83: Short write in ptask ============================================================== 07:04:31 (1544771071) [11566.114549] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ============================================= 07:04:33 (1544771073) [11569.948507] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 07:04:37 (1544771077) [11571.446846] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads =============================================== 07:04:38 (1544771078) [11600.616732] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 07:05:07 (1544771107) [11600.858584] Lustre: DEBUG MARKER: SKIP: sanity test_101b needs >= 2 OSTs [11602.028643] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ============================================ 07:05:09 (1544771109) [11612.901762] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================================== 07:05:19 (1544771119) [11740.654118] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ======================= 07:07:27 (1544771247) [11747.018001] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ===================================================== 07:07:34 (1544771254) [11753.549165] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ==================================================== 07:07:40 (1544771260) [11754.576168] Lustre: Unmounted lustre-client [11754.637578] Lustre: Mounted lustre-client [11766.609356] Lustre: Unmounted lustre-client [11766.669241] Lustre: Mounted lustre-client [11767.885655] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ================================================================= 07:07:54 (1544771274) [11769.169891] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs =========================================== 07:07:56 (1544771276) [11769.413298] Lustre: DEBUG MARKER: SKIP: sanity test_102b needs >= 2 OSTs [11770.577691] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs =================================== 07:07:57 (1544771277) [11770.816815] Lustre: DEBUG MARKER: SKIP: sanity test_102c needs >= 2 OSTs [11771.986998] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 07:07:59 (1544771279) [11772.230736] Lustre: DEBUG MARKER: SKIP: sanity test_102d needs >= 2 OSTs [11773.401673] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts =================================================== 07:08:00 (1544771280) [11773.637352] Lustre: DEBUG MARKER: SKIP: sanity test_102f needs >= 2 OSTs [11774.804894] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ================================== 07:08:01 (1544771281) [11775.064952] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [11775.316830] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [11775.569403] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [11775.821454] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [11776.995594] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ================================= 07:08:04 (1544771284) [11777.458827] Lustre: DEBUG MARKER: SKIP: sanity test_102ha large_xattr disabled [11778.630953] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ================================================= 07:08:05 (1544771285) [11779.810715] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 07:08:06 (1544771286) [11780.047856] Lustre: DEBUG MARKER: SKIP: sanity test_102j needs >= 2 OSTs [11781.213588] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ===================== 07:08:08 (1544771288) [11782.452061] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================= 07:08:09 (1544771289) [11783.644018] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 07:08:10 (1544771290) [11784.821093] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 07:08:11 (1544771291) [11786.637372] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ============================ 07:08:13 (1544771293) [11788.007916] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 07:08:15 (1544771295) [11789.365119] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values ======================================================= 07:08:16 (1544771296) [11790.783154] Lustre: DEBUG MARKER: == sanity test 103a: acl test ======================================================================== 07:08:17 (1544771297) [11800.799653] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ==================================================== 07:08:27 (1544771307) [11802.161618] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test ======================================================== 07:08:29 (1544771309) [11802.191637] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11802.201027] Lustre: lustre-OST0000-osc-ffff88007a3bd000: Connection to lustre-OST0000 (at 10.9.6.179@tcp) was lost; in progress operations using this service will wait for recovery to complete [11802.204346] LustreError: 167-0: lustre-OST0000-osc-ffff88007a3bd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [11802.206186] Lustre: lustre-OST0000-osc-ffff88007a3bd000: Connection restored to 10.9.6.179@tcp (at 10.9.6.179@tcp) [11802.207357] Lustre: Skipped 1 previous similar message [11802.538026] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88007a3bd000.ost_server_uuid 200 [11802.761972] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88007a3bd000.ost_server_uuid in FULL state after 0 sec [11803.970572] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 07:08:31 (1544771311) [11805.141083] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ======================================== 07:08:32 (1544771312) [11806.332548] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ======================================== 07:08:33 (1544771313) [11807.549634] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ======================================== 07:08:34 (1544771314) [11808.741782] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================== 07:08:35 (1544771315) [11808.767477] LustreError: 27044:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 315 sleeping for 10000ms [11818.767608] LustreError: 27044:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 315 awake [11819.930370] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ======================================== 07:08:47 (1544771327) [11821.116597] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 07:08:48 (1544771328) [11822.300088] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ================================================================== 07:08:49 (1544771329) [11824.513241] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ========================================================= 07:08:51 (1544771331) [11825.744068] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [11826.004957] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ================================================== 07:08:53 (1544771333) [11826.264449] Lustre: DEBUG MARKER: SKIP: sanity test_116a needs >= 2 OSTs [11827.455294] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ===================== 07:08:54 (1544771334) [11829.526644] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ================================================================ 07:08:56 (1544771336) [11830.732188] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============================================================= 07:08:57 (1544771337) [11832.706875] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 07:08:59 (1544771339) [11835.572319] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 07:09:02 (1544771342) [11837.497327] LustreError: 11-0: lustre-OST0000-osc-ffff88007a3bd000: operation ost_write to node 10.9.6.179@tcp failed: rc = -30 [11837.498614] LustreError: 23545:0:(osc_request.c:1681:osc_brw_redo_request()) @@@ redo for recoverable error -30 req@ffff88007b10e700 x1619806185686336/t0(0) o4->lustre-OST0000-osc-ffff88007a3bd000@10.9.6.179@tcp:6/4 lens 4704/192 e 0 to 0 dl 1544771354 ref 2 fl Interpret:RM/0/0 rc -30/-30 [11837.501234] LustreError: 23545:0:(osc_request.c:1681:osc_brw_redo_request()) Skipped 15 previous similar messages [11840.088583] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 07:09:07 (1544771347) [11867.760267] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 07:09:34 (1544771374) [11868.026886] Lustre: *** cfs_fail_loc=40a, val=0*** [11868.027475] Lustre: Skipped 29 previous similar messages [11868.028056] LustreError: 803:0:(osc_request.c:2030:osc_build_rpc()) prep_req failed: -22 [11868.028926] LustreError: 803:0:(osc_cache.c:2286:osc_check_rpcs()) Write request failed with -22 [11870.106878] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ===================================== 07:09:37 (1544771377) [11870.465628] LustreError: 1274:0:(osc_request.c:2030:osc_build_rpc()) prep_req failed: -12 [11870.466570] LustreError: 1274:0:(osc_cache.c:2286:osc_check_rpcs()) Write request failed with -12 [11872.693065] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ================================= 07:09:39 (1544771379) [11873.453571] LustreError: 11-0: lustre-OST0000-osc-ffff88007a3bd000: operation ost_write to node 10.9.6.179@tcp failed: rc = -5 [11874.455272] LustreError: 11-0: lustre-OST0000-osc-ffff88007a3bd000: operation ost_write to node 10.9.6.179@tcp failed: rc = -5 [11879.381916] LustreError: 11-0: lustre-OST0000-osc-ffff88007a3bd000: operation ost_write to node 10.9.6.179@tcp failed: rc = -5 [11879.383157] LustreError: Skipped 1 previous similar message [11883.384770] LustreError: 11-0: lustre-OST0000-osc-ffff88007a3bd000: operation ost_write to node 10.9.6.179@tcp failed: rc = -5 [11883.386005] LustreError: 23549:0:(osc_request.c:1806:brw_interpret()) lustre-OST0000-osc-ffff88007a3bd000: too many resent retries for object: 0:297978, rc = -5. [11883.387490] LustreError: 23549:0:(osc_request.c:1806:brw_interpret()) Skipped 1 previous similar message [11883.388506] Lustre: 23549:0:(llite_lib.c:2676:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.9.6.178@tcp:/lustre/fid: [0x200006993:0x4e0:0x0]// may get corrupted (rc -5) [11885.231931] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ================================== 07:09:52 (1544771392) [11893.597888] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error =========================================== 07:10:00 (1544771400) [11894.131022] LustreError: 11-0: lustre-OST0000-osc-ffff88007a3bd000: operation ost_write to node 10.9.6.179@tcp failed: rc = -14 [11894.132281] LustreError: Skipped 3 previous similar messages [11896.032400] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 07:10:03 (1544771403) [11905.306275] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir ======================================================================= 07:10:12 (1544771412) [11906.582230] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =================================================================== 07:10:13 (1544771413) [11907.858407] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ================================== 07:10:14 (1544771414) [11921.497054] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ============================== 07:10:28 (1544771428) [11922.812527] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ============================= 07:10:29 (1544771429) [11923.050945] Lustre: DEBUG MARKER: SKIP: sanity test_119b needs >= 2 OSTs [11924.224160] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ============================================ 07:10:31 (1544771431) [11925.596555] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 07:10:32 (1544771432) [11927.020043] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [11930.219297] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 07:10:37 (1544771437) [11931.823876] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ================================================== 07:10:38 (1544771438) [11933.386351] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ==================================================== 07:10:40 (1544771440) [11934.964794] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ================================================= 07:10:42 (1544771442) [11936.544420] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ================================================== 07:10:43 (1544771443) [11945.136736] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ================================================== 07:10:52 (1544771452) [11953.747406] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ============================================= 07:11:00 (1544771460) [11981.545498] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ================================================================= 07:11:28 (1544771488) [11981.591739] Lustre: *** cfs_fail_loc=310, val=0*** [11982.796237] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work =========================================================== 07:11:29 (1544771489) [11983.324404] Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec [11983.669287] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [11986.370033] Lustre: DEBUG MARKER: ls 1000 files without statahead: 1 sec [11987.143605] Lustre: DEBUG MARKER: ls 1000 files with statahead: 1 sec [12011.076113] Lustre: DEBUG MARKER: ls 10000 files without statahead: 13 sec [12015.392358] Lustre: DEBUG MARKER: ls 10000 files with statahead: 4 sec [12015.644712] Lustre: DEBUG MARKER: ls done [12023.511624] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 8 seconds [12023.762318] Lustre: DEBUG MARKER: rm done [12024.955206] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) =================== 07:12:12 (1544771532) [12026.432732] Lustre: *** cfs_fail_loc=803, val=0*** [12026.433503] LustreError: 23547:0:(mdc_locks.c:1240:mdc_intent_getattr_async_interpret()) ldlm_cli_enqueue_fini: -110 [12027.856459] Lustre: DEBUG MARKER: ls done [12030.222270] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ====================================================================== 07:12:17 (1544771537) [12030.467492] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [12033.980930] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff88007a3bd000 [12034.230685] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff88007a3bd000 [12034.474728] Lustre: DEBUG MARKER: LRU=2003 [12034.717553] Lustre: DEBUG MARKER: LIMIT=91899 [12034.958724] Lustre: DEBUG MARKER: LVF=55056 [12035.205233] Lustre: DEBUG MARKER: OLD_LVF=1 [12035.447129] Lustre: DEBUG MARKER: Sleep 50 sec [12085.733333] Lustre: DEBUG MARKER: Dropped 606 locks in 50s [12085.975669] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [12088.960731] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) =================================================== 07:13:16 (1544771596) [12094.587039] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [12102.149605] Lustre: DEBUG MARKER: ls -la time: 8 seconds [12102.398186] Lustre: DEBUG MARKER: lru_size = 200 [12113.355382] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [12118.284052] Lustre: DEBUG MARKER: ls -la time: 5 seconds [12118.529441] Lustre: DEBUG MARKER: lru_size = 4006 [12118.771084] Lustre: DEBUG MARKER: ls -la is 37% faster with lru resize enabled [12122.828961] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ===================================================== 07:13:49 (1544771629) [12144.360982] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs =============== 07:14:11 (1544771651) [12145.594898] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ================ 07:14:12 (1544771652) [12146.793414] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ================================================ 07:14:13 (1544771653) [12148.103766] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================== 07:14:15 (1544771655) [12149.314735] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================= 07:14:16 (1544771656) [12150.520799] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ======================================================== 07:14:17 (1544771657) [12192.137719] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ========================================================== 07:14:59 (1544771699) [12193.389148] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (1-stripe file) ========================================================== 07:15:00 (1544771700) [12193.636331] Lustre: DEBUG MARKER: SKIP: sanity test_130b needs >= 2 OSTs [12194.840593] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ================================================ 07:15:01 (1544771701) [12195.079339] Lustre: DEBUG MARKER: SKIP: sanity test_130c needs >= 2 OSTs [12196.274732] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ========================================================== 07:15:03 (1544771703) [12196.519868] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [12197.712497] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 07:15:04 (1544771704) [12197.952902] Lustre: DEBUG MARKER: SKIP: sanity test_130e needs >= 2 OSTs [12199.143191] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ========================================================= 07:15:06 (1544771706) [12200.345240] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ============================ 07:15:07 (1544771707) [12201.660972] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ============================================================== 07:15:08 (1544771708) [12202.980588] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ============================================= 07:15:10 (1544771710) [12204.171454] Lustre: DEBUG MARKER: == sanity test 131d: test short read ================================================================= 07:15:11 (1544771711) [12205.386263] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 07:15:12 (1544771712) [12206.621232] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ============================================================= 07:15:13 (1544771713) [12210.179552] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ======================================================= 07:15:17 (1544771717) [12212.654189] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ============================================================= 07:15:19 (1544771719) [12236.805217] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================================== 07:15:43 (1544771743) [12240.912634] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats ====================================== 07:15:48 (1544771748) [12243.602430] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ================= 07:15:50 (1544771750) [12244.250569] Lustre: 2486:0:(lprocfs_status.c:2493:lprocfs_wr_root_squash()) lustre: failed to set root_squash due to bad address, rc = -14 [12244.251981] Lustre: 2486:0:(lprocfs_status.c:2489:lprocfs_wr_root_squash()) lustre: failed to set root_squash to "", needs uid:gid format, rc = -22 [12244.257723] Lustre: 2491:0:(lprocfs_status.c:2561:lprocfs_wr_nosquash_nids()) lustre: failed to set nosquash_nids to "", bad address rc = -14 [12244.259097] Lustre: 2491:0:(lprocfs_status.c:2565:lprocfs_wr_nosquash_nids()) lustre: failed to set nosquash_nids due to string too long rc = -22 [12244.369376] LustreError: 2509:0:(gss_cli_upcall.c:240:gss_do_ctx_init_rpc()) ioctl size 5, expect 80, please check lgss_keyring version [12244.400215] LustreError: 2545:0:(ldlm_resource.c:365:lru_size_store()) lru_size: invalid value written [12244.462169] Lustre: 2621:0:(libcfs_string.c:129:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [12245.657817] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ================= 07:15:52 (1544771752) [12272.278797] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12272.519176] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12273.621905] Lustre: 23549:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544771773/real 1544771773] req@ffff880004d58c00 x1619806189186960/t0(0) o400->lustre-MDT0000-mdc-ffff88007a3bd000@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544771780 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [12273.625015] Lustre: lustre-MDT0000-mdc-ffff88007a3bd000: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [12273.626799] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [12273.635942] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xc586cced9dd3fc8 to 0xc586cceda166bd3 [12273.639281] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [12273.919053] Lustre: Unmounted lustre-client [12288.889463] Key type lgssc unregistered [12288.890030] Lustre: 4305:0:(gss_mech_switch.c:80:lgss_mech_unregister()) Unregister krb5 mechanism [12290.906661] LNet: Removed LNI 10.9.6.160@tcp [12291.114406] LNet: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 [12291.120584] alg: No test for adler32 (adler32-zlib) [12291.121206] alg: No test for crc32 (crc32-table) [12291.924009] Lustre: Lustre: Build Version: 2.11.0 [12291.958238] LNet: Added LNI 10.9.6.160@tcp [8/256/0/180] [12291.958960] LNet: Accept secure, port 988 [12293.473463] Lustre: 4471:0:(gss_svc_upcall.c:1185:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [12293.475221] Lustre: 4471:0:(gss_mech_switch.c:71:lgss_mech_register()) Register gssnull mechanism [12293.476175] Key type lgssc registered [12293.580832] Lustre: Echo OBD driver; http://www.lustre.org/ [12299.655063] Lustre: Mounted lustre-client [12301.326392] Lustre: DEBUG MARKER: Using TIMEOUT=100 [12307.726720] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ============================================= 07:16:54 (1544771814) [12311.700708] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 07:16:58 (1544771818) [12326.575291] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 07:17:13 (1544771833) [12432.645630] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 07:18:59 (1544771939) [12435.107865] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 07:19:02 (1544771942) [12435.408527] Lustre: Unmounted lustre-client [12435.482575] Lustre: Mounted lustre-client [12436.884890] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls =================================================== 07:19:04 (1544771944) [12441.014828] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================== 07:19:08 (1544771948) [12443.033325] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================= 07:19:10 (1544771950) [12444.338045] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 07:19:11 (1544771951) [12445.727523] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ================================================ 07:19:12 (1544771952) [12447.129600] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ===================================================================== 07:19:14 (1544771954) [12449.170900] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ================================================ 07:19:16 (1544771956) [12449.594755] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [12450.795427] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 07:19:17 (1544771957) [12452.204821] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ============================================================ 07:19:19 (1544771959) [12453.816334] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir ============================================== 07:19:20 (1544771960) [12455.201284] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea ============================================== 07:19:22 (1544771962) [12456.543322] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ========================================================= 07:19:23 (1544771963) [12501.659246] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 07:20:08 (1544772008) [12504.031217] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 07:20:11 (1544772011) [12506.393362] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 07:20:13 (1544772013) [12508.746842] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 07:20:15 (1544772015) [12511.110349] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 07:20:18 (1544772018) [12518.558422] Lustre: lfs: using old ioctl(LL_IOC_LOV_GETSTRIPE) on [0x200007932:0x882:0x0], use llapi_layout_get_by_path() [12521.035637] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 07:20:28 (1544772028) [12530.757430] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 07:20:37 (1544772037) [12540.340702] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 07:20:47 (1544772047) [12551.187734] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ========================================================= 07:20:58 (1544772058) [12552.581836] Lustre: DEBUG MARKER: Turn on read and write cache [12553.275728] Lustre: DEBUG MARKER: Write data and read it back. [12553.520158] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [12554.224439] Lustre: DEBUG MARKER: cache hits:: before: 8228, after: 8231 [12554.466553] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [12554.945912] Lustre: DEBUG MARKER: cache hits:: before: 8231, after: 8234 [12555.189147] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [12555.878461] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [12556.584875] Lustre: DEBUG MARKER: cache hits:: before: 8234, after: 8237 [12556.830693] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [12557.314338] Lustre: DEBUG MARKER: cache hits:: before: 8237, after: 8237 [12557.558206] Lustre: DEBUG MARKER: Write data and read it back. [12557.805145] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [12558.523470] Lustre: DEBUG MARKER: cache hits:: before: 8237, after: 8240 [12558.769110] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [12559.254807] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8240 [12559.503430] Lustre: DEBUG MARKER: Turn off read and write cache [12560.206596] Lustre: DEBUG MARKER: Write data and read it back [12560.452123] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [12561.181749] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8240 [12561.428728] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [12562.142992] Lustre: DEBUG MARKER: Write data and read it back [12562.388794] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [12563.161892] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8240 [12563.410884] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [12564.121997] Lustre: DEBUG MARKER: cache hits:: before: 8240, after: 8243 [12565.666698] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ================================================================ 07:21:12 (1544772072) [12566.608482] Lustre: DEBUG MARKER: sanity test_160a: @@@@@@ FAIL: User not found in changelog_users [12577.533654] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================= 07:21:24 (1544772084) [12578.486614] Lustre: DEBUG MARKER: sanity test_160b: @@@@@@ FAIL: User '' not found in changelog_users [12586.707860] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ============================== 07:21:33 (1544772093) [12589.420888] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event =============================== 07:21:36 (1544772096) [12589.699759] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [12590.923763] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) =============================== 07:21:38 (1544772098) [12591.697173] Lustre: DEBUG MARKER: sanity test_160e: @@@@@@ FAIL: changelog_deregister failed with 1, expect 2 (ENOENT) [12599.582618] Lustre: DEBUG MARKER: SKIP: sanity test_160f skipping ALWAYS excluded test 160f [12599.872276] Lustre: DEBUG MARKER: SKIP: sanity test_160g skipping ALWAYS excluded test 160g [12600.135889] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ================================================================== 07:21:47 (1544772107) [12604.142423] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory =========================================== 07:21:51 (1544772111) [12604.384226] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [12605.582330] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 07:21:52 (1544772112) [12608.027082] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ======================================= 07:21:55 (1544772115) [12608.409040] LustreError: 32685:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 140c sleeping for 5000ms [12613.408885] LustreError: 32685:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 140c awake [12615.109196] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ============================================================== 07:22:02 (1544772122) [12616.415524] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ============================================ 07:22:03 (1544772123) [12616.657722] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [12617.864778] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 07:22:05 (1544772125) [12621.561519] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock =================================== 07:22:08 (1544772128) [12621.807626] Lustre: DEBUG MARKER: creating a 10 Mb file [12622.448354] Lustre: DEBUG MARKER: starting reads [12622.814222] Lustre: DEBUG MARKER: truncating the file [12623.067278] Lustre: DEBUG MARKER: killing dd [12623.311553] Lustre: DEBUG MARKER: removing the temporary file [12624.501095] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ============================================= 07:22:11 (1544772131) [12624.517469] Lustre: debug daemon will attempt to start writing to /tmp/f170.sanity_log_good (512000kB max) [12624.524530] Lustre: shutting down debug daemon thread... [12625.522417] Lustre: debug daemon will attempt to start writing to /tmp/f170.sanity_log_good (512000kB max) [12625.528889] Lustre: shutting down debug daemon thread... [12627.766436] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ============================== 07:22:14 (1544772134) [12627.787699] LustreError: 4824:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 50e sleeping for 3000ms [12629.354959] Lustre: *** cfs_fail_loc=50e, val=0*** [12629.355609] LustreError: dumping log to /tmp/lustre-log.1544772136.8339 [12630.788583] LustreError: 4824:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 50e awake [12630.789668] LustreError: dumping log to /tmp/lustre-log.1544772138.4824 [12631.997206] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ============================================================= 07:22:19 (1544772139) [12633.918135] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter ============================================== 07:22:21 (1544772141) [12637.567424] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ============================= 07:22:24 (1544772144) [12642.195501] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir =========================================================== 07:22:29 (1544772149) [12660.248102] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================= 07:22:47 (1544772167) [12670.909409] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ========================= 07:22:58 (1544772178) [12672.748168] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =============================================================== 07:22:59 (1544772179) [12674.500469] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ==================================== 07:23:01 (1544772181) [12675.778587] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ================================================ 07:23:02 (1544772182) [12684.051311] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap =================================================== 07:23:11 (1544772191) [12685.471366] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps =================================== 07:23:12 (1544772192) [12686.924952] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ============================================================ 07:23:14 (1544772194) [12689.354104] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ======================================================== 07:23:16 (1544772196) [12691.402027] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ======================================= 07:23:18 (1544772198) [12692.974458] Lustre: DEBUG MARKER: == sanity test 200: OST pools ======================================================================== 07:23:20 (1544772200) [12731.736323] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ================================================= 07:23:58 (1544772238) [12732.981141] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ============================================ 07:24:00 (1544772240) [12734.215753] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset =========================================== 07:24:01 (1544772241) [12735.445344] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ============================================= 07:24:02 (1544772242) [12736.672461] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ===================================================== 07:24:03 (1544772243) [12737.902678] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ================================================ 07:24:05 (1544772245) [12739.133892] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset =============================================== 07:24:06 (1544772246) [12740.368822] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ================================================= 07:24:07 (1544772247) [12741.606166] Lustre: DEBUG MARKER: == sanity test 205: Verify job stats ================================================================= 07:24:08 (1544772248) [12753.082045] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity [12753.333337] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.27215 [12753.919474] Lustre: DEBUG MARKER: sanity test_205: @@@@@@ FAIL: No jobstats for id.205.mkdir.27215 found on mds::*..job_stats [12764.781109] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug =============================================== 07:24:31 (1544772271) [12766.011051] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse =================================================== 07:24:33 (1544772273) [12767.239700] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ====================================================== 07:24:34 (1544772274) [12768.502371] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open =================================================================== 07:24:35 (1544772275) [12793.262035] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12793.509965] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12794.677953] LustreError: 11-0: lustre-MDT0000-mdc-ffff880062b69800: operation ldlm_enqueue to node 10.9.6.178@tcp failed: rc = -107 [12794.679283] Lustre: lustre-MDT0000-mdc-ffff880062b69800: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [12794.685500] LustreError: 4455:0:(client.c:2972:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800682c5e00 x1619810635740384/t68719554866(68719554866) o101->lustre-MDT0000-mdc-ffff880062b69800@10.9.6.178@tcp:12/10 lens 808/560 e 0 to 0 dl 1544772345 ref 2 fl Interpret:RP/4/0 rc 301/301 [12794.728533] Lustre: lustre-MDT0000-mdc-ffff880062b69800: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [12814.098928] Lustre: 4458:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544772314/real 1544772314] req@ffff88003e92cf00 x1619810635740768/t0(0) o400->MGC10.9.6.178@tcp@10.9.6.178@tcp:26/25 lens 224/224 e 0 to 1 dl 1544772321 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [12814.101933] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [12815.964506] Lustre: DEBUG MARKER: trevis-60vm7.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12816.207620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12817.371884] LustreError: 11-0: lustre-MDT0000-mdc-ffff880062b69800: operation mds_statfs to node 10.9.6.178@tcp failed: rc = -107 [12817.373179] Lustre: lustre-MDT0000-mdc-ffff880062b69800: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [12817.379251] LustreError: 4455:0:(client.c:2972:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800682c7c00 x1619810635740704/t73014444035(73014444035) o101->lustre-MDT0000-mdc-ffff880062b69800@10.9.6.178@tcp:12/10 lens 816/560 e 0 to 0 dl 1544772366 ref 2 fl Interpret:RP/4/0 rc 301/301 [12817.382065] LustreError: 4455:0:(client.c:2972:ptlrpc_replay_interpret()) Skipped 1 previous similar message [12817.460219] Lustre: lustre-MDT0000-mdc-ffff880062b69800: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [12818.691865] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly =========================== 07:25:25 (1544772325) [12824.077476] bash (21150): drop_caches: 3 [12825.276424] bash (21150): drop_caches: 3 [12826.268006] Lustre: 4458:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544772289/real 1544772289] req@ffff88003e92c300 x1619810635740528/t0(0) o400->lustre-MDT0000-mdc-ffff880062b69800@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544772333 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [12826.561866] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ==================================================================== 07:25:33 (1544772333) [12828.292077] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ====================== 07:25:35 (1544772335) [12828.321432] LustreError: 4456:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 40f sleeping for 10000ms [12828.324293] LustreError: 4456:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 40f awake [12829.521767] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 07:25:36 (1544772336) [12832.281636] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ==================== 07:25:39 (1544772339) [12833.563400] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ================== 07:25:40 (1544772340) [12836.421560] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ================================== 07:25:43 (1544772343) [12838.078361] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock =================================== 07:25:45 (1544772345) [12838.323825] Lustre: DEBUG MARKER: creating a 10 Mb file [12838.981137] Lustre: DEBUG MARKER: starting reads [12839.105177] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xc586cceda1906a8 to 0xc586cceda3693b7 [12839.108158] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [12839.337066] Lustre: DEBUG MARKER: truncating the file [12839.601838] Lustre: DEBUG MARKER: killing dd [12839.850801] Lustre: DEBUG MARKER: removing the temporary file [12841.079730] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ================== 07:25:48 (1544772348) [12841.096426] Lustre: *** cfs_fail_loc=411, val=0*** [12842.471135] Lustre: DEBUG MARKER: SKIP: sanity test_220 skipping excluded test 220 [12842.744274] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM =============================== 07:25:49 (1544772349) [12847.667726] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================================= 07:25:54 (1544772354) [12848.943218] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ============================== 07:25:56 (1544772356) [12850.001436] Lustre: 4458:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544772314/real 1544772314] req@ffff88003e92de00 x1619810635740784/t0(0) o400->lustre-MDT0000-mdc-ffff880062b69800@10.9.6.178@tcp:12/10 lens 224/224 e 0 to 1 dl 1544772357 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [12850.224475] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ======================================== 07:25:57 (1544772357) [12851.492850] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ================================================== 07:25:58 (1544772358) [12852.811334] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ================================================== 07:25:59 (1544772359) [12854.046053] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ============================== 07:26:01 (1544772361) [12889.117290] Lustre: lustre-OST0000-osc-ffff880062b69800: Connection to lustre-OST0000 (at 10.9.6.179@tcp) was lost; in progress operations using this service will wait for recovery to complete [12889.126344] LustreError: 167-0: lustre-OST0000-osc-ffff880062b69800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [12889.132304] Lustre: lustre-OST0000-osc-ffff880062b69800: Connection restored to 10.9.6.179@tcp (at 10.9.6.179@tcp) [12890.197147] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a [12890.502552] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b [12890.773710] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ================================= 07:26:37 (1544772397) [12892.105244] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ================ 07:26:39 (1544772399) [12892.349228] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [12893.561135] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ================================== 07:26:40 (1544772400) [12894.826444] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ===================================================== 07:26:41 (1544772401) [12894.877767] Lustre: *** cfs_fail_loc=1002, val=0*** [12937.896273] Lustre: DEBUG MARKER: SKIP: sanity test_228b skipping excluded test 228b [12938.167962] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ================= 07:27:25 (1544772445) [12938.221700] Lustre: *** cfs_fail_loc=1002, val=0*** [13007.233385] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ============================ 07:28:34 (1544772514) [13007.665703] Lustre: DEBUG MARKER: SKIP: sanity test_229 needs >= 2 OSTs [13008.883991] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ==================== 07:28:36 (1544772516) [13009.127320] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [13010.345973] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =============================================================== 07:28:37 (1544772517) [13010.589848] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [13011.796685] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration is failed ============================= 07:28:38 (1544772518) [13012.066708] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [13013.276073] Lustre: DEBUG MARKER: == sanity test 230d: check migrate big directory ===================================================== 07:28:40 (1544772520) [13013.520949] Lustre: DEBUG MARKER: SKIP: sanity test_230d needs >= 2 MDTs [13014.752750] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files =============================================== 07:28:41 (1544772521) [13015.017791] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [13016.235223] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files ============================================== 07:28:43 (1544772523) [13016.478262] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [13017.700694] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ==================================================== 07:28:44 (1544772524) [13017.951337] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [13019.175055] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ============================================================= 07:28:46 (1544772526) [13019.425853] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [13020.644657] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ======================================= 07:28:47 (1544772527) [13020.892189] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [13022.113679] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ================ 07:28:49 (1544772529) [13025.382249] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ============================ 07:28:52 (1544772532) [13030.088124] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ============================================= 07:28:57 (1544772537) [13030.274904] LustreError: 11-0: lustre-OST0000-osc-ffff880062b69800: operation ldlm_enqueue to node 10.9.6.179@tcp failed: rc = -12 [13030.824125] Lustre: Unmounted lustre-client [13030.888164] Lustre: Mounted lustre-client [13033.170783] Lustre: DEBUG MARKER: sanity test_232a: @@@@@@ FAIL: cannot start ost1 [13044.492277] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ================================ 07:29:11 (1544772551) [13046.797670] Lustre: Unmounted lustre-client [13046.859904] Lustre: Mounted lustre-client [13048.916865] Lustre: DEBUG MARKER: sanity test_232b: @@@@@@ FAIL: cannot start ost1 [13056.496864] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ======================================= 07:29:23 (1544772563) [13057.948576] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ==================================== 07:29:25 (1544772565) [13059.426872] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM =========================================== 07:29:26 (1544772566) [13059.489754] Lustre: *** cfs_fail_loc=1405, val=0*** [13060.757460] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================= 07:29:27 (1544772567) [13064.153132] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ================================================ 07:29:31 (1544772571) [13065.452999] Lustre: DEBUG MARKER: == sanity test 237: Verify name_to_handle_at/open_by_handle_at syscalls ============================== 07:29:32 (1544772572) [13066.675284] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ======================================================== 07:29:33 (1544772573) [13068.103053] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test =================================================================== 07:29:35 (1544772575) [13086.214370] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ======================================= 07:29:53 (1544772593) [13097.647814] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ============================= 07:30:04 (1544772604) [13101.737784] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ===================== 07:30:08 (1544772608) [13101.989436] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [13103.196950] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ====================================================================== 07:30:10 (1544772610) [13114.030072] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ====================================================================== 07:30:21 (1544772621) [13120.878556] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ======================= 07:30:28 (1544772628) [13121.086045] LustreError: 11-0: lustre-MDT0000-mdc-ffff880076036800: operation mds_readpage to node 10.9.6.178@tcp failed: rc = -12 [13122.459102] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ========================================================= 07:30:29 (1544772629) [13123.000990] Lustre: 25001:0:(file.c:2046:ll_get_grouplock()) group lock already existed with gid 97486 [13123.002114] Lustre: 25001:0:(file.c:2115:ll_put_grouplock()) no group lock held [13123.002900] Lustre: 25001:0:(file.c:2036:ll_get_grouplock()) group id for group lock must not be 0 [13123.004452] Lustre: 25001:0:(file.c:2123:ll_put_grouplock()) group lock 4294967286 doesn't match current id 3543 [13138.206447] Lustre: 25001:0:(file.c:2115:ll_put_grouplock()) no group lock held [13138.230546] Lustre: 25001:0:(file.c:2036:ll_get_grouplock()) group id for group lock must not be 0 [13139.459717] Lustre: DEBUG MARKER: == sanity test 244: sendfile with group lock tests =================================================== 07:30:46 (1544772646) [13155.953901] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ============================= 07:31:03 (1544772663) [13157.178440] Lustre: DEBUG MARKER: == sanity test 246: Read file of size 4095 should return right length ================================ 07:31:04 (1544772664) [13158.948922] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ========================================================= 07:31:06 (1544772666) [13159.024507] Lustre: Mounted lustre-client [13159.088557] Lustre: Unmounted lustre-client [13160.747265] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ================================================ 07:31:07 (1544772667) [13160.807680] LustreError: 27411:0:(llite_lib.c:503:client_common_fill_super()) cannot mds_connect: rc = -2 [13160.826103] LustreError: 27411:0:(obd_mount.c:1583:lustre_fill_super()) Unable to mount (-2) [13162.027909] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside root =================================================== 07:31:09 (1544772669) [13163.647382] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside root ==================================================== 07:31:10 (1544772670) [13163.780356] Lustre: Unmounted lustre-client [13163.780878] Lustre: Skipped 4 previous similar messages [13165.342195] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ============================================================= 07:31:12 (1544772672) [13165.508548] LustreError: 11-0: lustre-MDT0000-mdc-ffff88003df94000: operation mds_get_root to node 10.9.6.178@tcp failed: rc = -22 [13165.509945] LustreError: 28992:0:(llite_lib.c:503:client_common_fill_super()) cannot mds_connect: rc = -22 [13165.527406] LustreError: 28992:0:(obd_mount.c:1583:lustre_fill_super()) Unable to mount (-22) [13166.736020] Lustre: DEBUG MARKER: == sanity test 248: fast read verification =========================================================== 07:31:13 (1544772673) [13177.504276] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ========================================================= 07:31:24 (1544772684) [13178.906349] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ============================================================ 07:31:26 (1544772686) [13180.135344] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================== 07:31:27 (1544772687) [13180.159618] Lustre: *** cfs_fail_loc=1407, val=0*** [13181.406495] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ============================================================= 07:31:28 (1544772688) [13183.668721] Lustre: DEBUG MARKER: SKIP: sanity test_253 skipping ALWAYS excluded test 253 [13183.942374] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ============================================================= 07:31:31 (1544772691) [13184.375766] Lustre: DEBUG MARKER: SKIP: sanity test_254 MDS does not support changelog_size [13185.592162] Lustre: DEBUG MARKER: == sanity test 255a: check 'lfs ladvise -a willread' ================================================= 07:31:32 (1544772692) [13585.446780] Lustre: DEBUG MARKER: == sanity test 255b: check 'lfs ladvise -a dontneed' ================================================= 07:38:12 (1544773092) [13591.855585] Lustre: DEBUG MARKER: == sanity test 255c: suite of ladvise lockahead tests ================================================ 07:38:19 (1544773099) [13595.175037] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state =================================== 07:38:22 (1544773102) [13598.478914] Lustre: DEBUG MARKER: sanity test_256: @@@@@@ FAIL: Fail to start MDT [13609.477277] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ========================================================= 07:38:36 (1544773116) [13609.916311] LustreError: 11-0: lustre-MDT0000-mdc-ffff880076036800: operation ldlm_enqueue to node 10.9.6.178@tcp failed: rc = -14 [13615.473724] Lustre: lustre-MDT0000-mdc-ffff880076036800: Connection to lustre-MDT0000 (at 10.9.6.178@tcp) was lost; in progress operations using this service will wait for recovery to complete [13621.219848] Lustre: DEBUG MARKER: == sanity test 258a: ================================================================================= 07:38:48 (1544773128) [13622.474449] Lustre: 4460:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1544773122/real 1544773122] req@ffff88006a8f7600 x1619810654976192/t0(0) o400->MGC10.9.6.178@tcp@10.9.6.178@tcp:26/25 lens 224/224 e 0 to 1 dl 1544773129 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13622.477260] LustreError: 166-1: MGC10.9.6.178@tcp: Connection to MGS (at 10.9.6.178@tcp) was lost; in progress operations using this service will fail [13622.479333] Lustre: Evicted from MGS (at 10.9.6.178@tcp) after server handle changed from 0xc586cceda62cbc8 to 0xc586cceda6dbb35 [13622.481213] Lustre: MGC10.9.6.178@tcp: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [13632.385553] Lustre: lustre-MDT0000-mdc-ffff880076036800: Connection restored to 10.9.6.178@tcp (at 10.9.6.178@tcp) [13633.670677] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ================================================ 07:39:00 (1544773140) [13635.433604] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ============================================================= 07:39:02 (1544773142) [13635.448445] Lustre: *** cfs_fail_loc=806, val=0*** [13635.448998] Lustre: 5230:0:(mdc_request.c:856:mdc_close()) lustre-MDT0000-mdc-ffff880076036800: close of FID [0x2000088d4:0x13d6:0x0] failed, file reference will be dropped when this client unmounts or is evicted [13635.450933] LustreError: 5230:0:(file.c:205:ll_close_inode_openhandle()) lustre-clilmv-ffff880076036800: inode [0x2000088d4:0x13d6:0x0] mdc close failed: rc = -12 [13636.659918] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ================================================== 07:39:03 (1544773143) [13641.031375] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ============================= 07:39:08 (1544773148) [13642.511650] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests =================================================== 07:39:09 (1544773149) [13643.960683] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================== 07:39:11 (1544773151) [13645.396534] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test =============================================== 07:39:12 (1544773152) [13646.948706] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ============================================= 07:39:14 (1544773154) [13647.882779] Lustre: DEBUG MARKER: sanity test_270f: @@@@@@ FAIL: Able to create DoM component size more than LOD limit