[    0.000000] Linux version 5.14.0-503.40.1.el9_5.x86_64 (mockbuild@x86-64-01.build.eng.rdu2.redhat.com) (gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5), GNU ld version 2.35.2-54.el9) #1 SMP PREEMPT_DYNAMIC Thu Apr 24 08:27:29 EDT 2025
[    0.000000] The list of certified hardware and cloud instances for Red Hat Enterprise Linux 9 can be viewed at the Red Hat Ecosystem Catalog, https://catalog.redhat.com.
[    0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-5.14.0-503.40.1.el9_5.x86_64 root=UUID=83f639ed-9bac-410b-92f1-8e2ef103e35e ro resume=UUID=dca934f6-a33b-47f1-8763-afe07d62039b selinux=0 console=ttyS0,115200 console=tty0 console=ttyS0,115200 net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M slab_nomerge slub_nomerge
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bfff9fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bfffa000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] APIC: Static calls initialized
[    0.000000] SMBIOS 2.4 present.
[    0.000000] DMI: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    0.000000] Hypervisor detected: KVM
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000001] kvm-clock: using sched offset of 820237919311 cycles
[    0.000010] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000027] tsc: Detected 2593.748 MHz processor
[    0.000376] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000379] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000384] last_pfn = 0x140000 max_arch_pfn = 0x400000000
[    0.000425] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs
[    0.000439] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.000493] last_pfn = 0xbfffa max_arch_pfn = 0x400000000
[    0.015039] found SMP MP-table at [mem 0x000f6390-0x000f639f]
[    0.015095] Using GB pages for direct mapping
[    0.015213] RAMDISK: [mem 0x34410000-0x361fffff]
[    0.015244] ACPI: Early table checksum verification disabled
[    0.015281] ACPI: RSDP 0x00000000000F61A0 000014 (v00 BOCHS )
[    0.015296] ACPI: RSDT 0x00000000BFFFFB40 000030 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
[    0.015328] ACPI: FACP 0x00000000BFFFF177 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
[    0.015344] ACPI: DSDT 0x00000000BFFFE040 001137 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
[    0.015353] ACPI: FACS 0x00000000BFFFE000 000040
[    0.015359] ACPI: SSDT 0x00000000BFFFF1EB 0008D5 (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
[    0.015364] ACPI: APIC 0x00000000BFFFFAC0 000080 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
[    0.015368] ACPI: Reserving FACP table memory at [mem 0xbffff177-0xbffff1ea]
[    0.015370] ACPI: Reserving DSDT table memory at [mem 0xbfffe040-0xbffff176]
[    0.015371] ACPI: Reserving FACS table memory at [mem 0xbfffe000-0xbfffe03f]
[    0.015372] ACPI: Reserving SSDT table memory at [mem 0xbffff1eb-0xbffffabf]
[    0.015373] ACPI: Reserving APIC table memory at [mem 0xbffffac0-0xbffffb3f]
[    0.015897] No NUMA configuration found
[    0.015903] Faking a node at [mem 0x0000000000000000-0x000000013fffffff]
[    0.015917] NODE_DATA(0) allocated [mem 0x13ffd5000-0x13fffffff]
[    0.016419] Reserving 256MB of memory at 2800MB for crashkernel (System RAM: 4095MB)
[    0.016504] Zone ranges:
[    0.016511]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.016513]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.016518]   Normal   [mem 0x0000000100000000-0x000000013fffffff]
[    0.016520]   Device   empty
[    0.016524] Movable zone start for each node
[    0.016527] Early memory node ranges
[    0.016527]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.016529]   node   0: [mem 0x0000000000100000-0x00000000bfff9fff]
[    0.016530]   node   0: [mem 0x0000000100000000-0x000000013fffffff]
[    0.016532] Initmem setup node 0 [mem 0x0000000000001000-0x000000013fffffff]
[    0.016555] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.016785] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.062133] On node 0, zone Normal: 6 pages in unavailable ranges
[    0.062724] ACPI: PM-Timer IO Port: 0x608
[    0.062756] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.062808] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.062815] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.062818] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.062826] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.062834] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.062835] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.062843] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.062852] TSC deadline timer available
[    0.062854] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.062885] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()
[    0.062960] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.062965] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.062966] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.062967] PM: hibernation: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.062968] PM: hibernation: Registered nosave memory: [mem 0xbfffa000-0xbfffffff]
[    0.062970] PM: hibernation: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
[    0.062971] PM: hibernation: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
[    0.062971] PM: hibernation: Registered nosave memory: [mem 0xff000000-0xfffbffff]
[    0.062972] PM: hibernation: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
[    0.062980] [mem 0xc0000000-0xfeffbfff] available for PCI devices
[    0.062985] Booting paravirtualized kernel on KVM
[    0.062989] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.063001] Warning: Deprecated Hardware is detected: x86_64-v2:GenuineIntel:Intel Xeon E312xx (Sandy Bridge) will not be maintained in a future major release and may be disabled
[    0.070382] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.072388] percpu: Embedded 64 pages/cpu s225280 r8192 d28672 u1048576
[    0.072399] pcpu-alloc: s225280 r8192 d28672 u1048576 alloc=1*2097152
[    0.072401] pcpu-alloc: [0] 0 1 
[    0.072429] kvm-guest: PV spinlocks enabled
[    0.072436] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.072449] Kernel command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-5.14.0-503.40.1.el9_5.x86_64 root=UUID=83f639ed-9bac-410b-92f1-8e2ef103e35e ro resume=UUID=dca934f6-a33b-47f1-8763-afe07d62039b selinux=0 console=ttyS0,115200 console=tty0 console=ttyS0,115200 net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M slab_nomerge slub_nomerge
[    0.072874] Unknown kernel command line parameters "BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-5.14.0-503.40.1.el9_5.x86_64", will be passed to user space.
[    0.076092] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.077703] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.077821] Fallback order for Node 0: 0 
[    0.077828] Built 1 zonelists, mobility grouping on.  Total pages: 1031930
[    0.077833] Policy zone: Normal
[    0.079572] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.079609] software IO TLB: area num 2.
[    0.157713] Memory: 2917156K/4193888K available (16384K kernel code, 5685K rwdata, 12912K rodata, 3976K init, 5672K bss, 485496K reserved, 0K cma-reserved)
[    0.158144] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.158188] Kernel/User page tables isolation: enabled
[    0.158305] ftrace: allocating 48139 entries in 189 pages
[    0.172293] ftrace: allocated 189 pages with 6 groups
[    0.173162] Dynamic Preempt: voluntary
[    0.173284] rcu: Preemptible hierarchical RCU implementation.
[    0.173289] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=2.
[    0.173295] 	Trampoline variant of Tasks RCU enabled.
[    0.173298] 	Rude variant of Tasks RCU enabled.
[    0.173299] 	Tracing variant of Tasks RCU enabled.
[    0.173300] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[    0.173301] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.179480] NR_IRQS: 524544, nr_irqs: 440, preallocated irqs: 16
[    0.179719] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.179895] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)
[    0.180057] Console: colour *CGA 80x25
[    0.180061] printk: legacy console [tty0] enabled
[    0.186419] printk: console [ttyS0] enabled
[    0.334631] ACPI: Core revision 20230331
[    0.335586] APIC: Switch to symmetric I/O mode setup
[    0.336899] x2apic enabled
[    0.337884] APIC: Switched APIC routing to: physical x2apic
[    0.340419] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.341769] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x25632a14f90, max_idle_ns: 440795228339 ns
[    0.343985] Calibrating delay loop (skipped) preset value.. 5187.49 BogoMIPS (lpj=2593748)
[    0.346194] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[    0.347051] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
[    0.348047] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.350047] Spectre V2 : Mitigation: Retpolines
[    0.351043] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.353043] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.354044] Speculative Store Bypass: Vulnerable
[    0.355044] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    0.357043] MMIO Stale Data: Unknown: No mitigations
[    0.358057] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.359044] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.361042] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.362044] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.363983] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.381658] Freeing SMP alternatives memory: 40K
[    0.382046] pid_max: default: 32768 minimum: 301
[    0.383123] LSM: initializing lsm=lockdown,capability,yama,integrity,bpf
[    0.385054] Yama: becoming mindful.
[    0.386071] LSM support for eBPF active
[    0.387127] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.388101] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.391154] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (family: 0x6, model: 0x2a, stepping: 0x1)
[    0.392641] RCU Tasks: Setting shift to 1 and lim to 1 rcu_task_cb_adjust=1.
[    0.393119] RCU Tasks Rude: Setting shift to 1 and lim to 1 rcu_task_cb_adjust=1.
[    0.394116] RCU Tasks Trace: Setting shift to 1 and lim to 1 rcu_task_cb_adjust=1.
[    0.395095] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
[    0.396086] signal: max sigframe size: 1776
[    0.397154] rcu: Hierarchical SRCU implementation.
[    0.397221] rcu: 	Max phase no-delay instances is 400.
[    0.399698] NMI watchdog: Perf NMI watchdog permanently disabled
[    0.400195] smp: Bringing up secondary CPUs ...
[    0.401428] smpboot: x86: Booting SMP configuration:
[    0.401502] .... node  #0, CPUs:      #1
[    0.159253] smpboot: CPU 1 Converting physical 0 to logical die 1
[    0.404024] smp: Brought up 1 node, 2 CPUs
[    0.404212] smpboot: Max logical packages: 2
[    0.404276] smpboot: Total of 2 processors activated (10374.99 BogoMIPS)
[    0.414839] node 0 deferred pages initialised in 8ms
[    0.416521] devtmpfs: initialized
[    0.417093] x86/mm: Memory block size: 128MB
[    0.419093] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.420000] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.421098] pinctrl core: initialized pinctrl subsystem
[    0.423588] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.424190] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[    0.424994] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.425994] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.427020] audit: initializing netlink subsys (disabled)
[    0.428033] audit: type=2000 audit(1764877535.460:1): state=initialized audit_enabled=0 res=1
[    0.428357] thermal_sys: Registered thermal governor 'fair_share'
[    0.428360] thermal_sys: Registered thermal governor 'step_wise'
[    0.428466] thermal_sys: Registered thermal governor 'user_space'
[    0.430997] cpuidle: using governor menu
[    0.432721] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.432872] PCI: Using configuration type 1 for base access
[    0.436213] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.439224] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.439614] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.439614] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.439614] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    0.450228] cryptd: max_cpu_qlen set to 1000
[    0.451507] ACPI: Added _OSI(Module Device)
[    0.451990] ACPI: Added _OSI(Processor Device)
[    0.452886] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.453988] ACPI: Added _OSI(Processor Aggregator Device)
[    0.454989] ACPI: Added _OSI(Linux-Dell-Video)
[    0.455869] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.456996] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.459021] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    0.465060] ACPI: _OSC evaluation for CPUs failed, trying _PDC
[    0.466135] ACPI: Interpreter enabled
[    0.466917] ACPI: PM: (supports S0 S5)
[    0.467985] ACPI: Using IOAPIC for interrupt routing
[    0.468099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.468208] PCI: Using E820 reservations for host bridge windows
[    0.468699] ACPI: Enabled 16 GPEs in block 00 to 0F
[    0.471862] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.471980] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI EDR HPX-Type3]
[    0.471988] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]
[    0.472099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended configuration space under this bridge
[    0.472664] acpiphp: Slot [3] registered
[    0.472780] acpiphp: Slot [4] registered
[    0.472892] acpiphp: Slot [5] registered
[    0.473031] acpiphp: Slot [6] registered
[    0.473157] acpiphp: Slot [7] registered
[    0.473296] acpiphp: Slot [8] registered
[    0.473450] acpiphp: Slot [9] registered
[    0.473586] acpiphp: Slot [10] registered
[    0.473679] acpiphp: Slot [11] registered
[    0.473822] acpiphp: Slot [12] registered
[    0.473939] acpiphp: Slot [13] registered
[    0.474031] acpiphp: Slot [14] registered
[    0.474180] acpiphp: Slot [15] registered
[    0.474322] acpiphp: Slot [16] registered
[    0.474434] acpiphp: Slot [17] registered
[    0.474551] acpiphp: Slot [18] registered
[    0.474679] acpiphp: Slot [19] registered
[    0.474814] acpiphp: Slot [20] registered
[    0.474907] acpiphp: Slot [21] registered
[    0.475015] acpiphp: Slot [22] registered
[    0.475144] acpiphp: Slot [23] registered
[    0.475281] acpiphp: Slot [24] registered
[    0.475392] acpiphp: Slot [25] registered
[    0.475492] acpiphp: Slot [26] registered
[    0.475604] acpiphp: Slot [27] registered
[    0.475738] acpiphp: Slot [28] registered
[    0.475867] acpiphp: Slot [29] registered
[    0.475999] acpiphp: Slot [30] registered
[    0.476121] acpiphp: Slot [31] registered
[    0.476242] PCI host bridge to bus 0000:00
[    0.476308] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.476425] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.476532] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.476661] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
[    0.476777] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.476980] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 conventional PCI endpoint
[    0.477928] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 conventional PCI endpoint
[    0.479165] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 conventional PCI endpoint
[    0.484681] pci 0000:00:01.1: BAR 4 [io  0xc0a0-0xc0af]
[    0.486765] pci 0000:00:01.1: BAR 0 [io  0x01f0-0x01f7]: legacy IDE quirk
[    0.486857] pci 0000:00:01.1: BAR 1 [io  0x03f6]: legacy IDE quirk
[    0.486967] pci 0000:00:01.1: BAR 2 [io  0x0170-0x0177]: legacy IDE quirk
[    0.486986] pci 0000:00:01.1: BAR 3 [io  0x0376]: legacy IDE quirk
[    0.487373] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300 conventional PCI endpoint
[    0.492685] pci 0000:00:01.2: BAR 4 [io  0xc040-0xc05f]
[    0.495011] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 conventional PCI endpoint
[    0.495914] pci 0000:00:01.3: quirk: [io  0x0600-0x063f] claimed by PIIX4 ACPI
[    0.496003] pci 0000:00:01.3: quirk: [io  0x0700-0x070f] claimed by PIIX4 SMB
[    0.496613] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint
[    0.497991] pci 0000:00:03.0: BAR 0 [io  0xc060-0xc07f]
[    0.499987] pci 0000:00:03.0: BAR 1 [mem 0xfebc0000-0xfebc0fff]
[    0.509986] pci 0000:00:03.0: ROM [mem 0xfeb80000-0xfebbffff pref]
[    0.510987] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint
[    0.512986] pci 0000:00:04.0: BAR 0 [io  0xc000-0xc03f]
[    0.514986] pci 0000:00:04.0: BAR 1 [mem 0xfebc1000-0xfebc1fff]
[    0.525714] pci 0000:00:05.0: [1af4:1002] type 00 class 0x00ff00 conventional PCI endpoint
[    0.526690] pci 0000:00:05.0: BAR 0 [io  0xc080-0xc09f]
[    0.533753] ACPI: PCI: Interrupt link LNKA configured for IRQ 10
[    0.534004] ACPI: PCI: Interrupt link LNKB configured for IRQ 10
[    0.534229] ACPI: PCI: Interrupt link LNKC configured for IRQ 11
[    0.534490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11
[    0.534667] ACPI: PCI: Interrupt link LNKS configured for IRQ 9
[    0.535074] iommu: Default domain type: Translated
[    0.535179] iommu: DMA domain TLB invalidation policy: lazy mode
[    0.535452] SCSI subsystem initialized
[    0.535606] ACPI: bus type USB registered
[    0.535721] usbcore: registered new interface driver usbfs
[    0.535813] usbcore: registered new interface driver hub
[    0.535933] usbcore: registered new device driver usb
[    0.536050] pps_core: LinuxPPS API ver. 1 registered
[    0.536151] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.536263] PTP clock support registered
[    0.536508] EDAC MC: Ver: 3.0.0
[    0.537339] NetLabel: Initializing
[    0.537444] NetLabel:  domain hash size = 128
[    0.537548] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.537692] NetLabel:  unlabeled traffic allowed by default
[    0.537799] PCI: Using ACPI for IRQ routing
[    0.537878] PCI: pci_cache_line_size set to 64 bytes
[    0.538149] e820: reserve RAM buffer [mem 0x0009f800-0x0009ffff]
[    0.538153] e820: reserve RAM buffer [mem 0xbfffa000-0xbfffffff]
[    0.542000] vgaarb: loaded
[    0.542690] clocksource: Switched to clocksource kvm-clock
[    0.542980] VFS: Disk quotas dquot_6.6.0
[    0.542980] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.542980] pnp: PnP ACPI init
[    0.543087] pnp 00:03: [dma 2]
[    0.543518] pnp: PnP ACPI: found 5 devices
[    0.558627] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.558911] NET: Registered PF_INET protocol family
[    0.565742] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.567041] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    0.567207] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.567295] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.567627] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    0.568149] TCP: Hash tables configured (established 32768 bind 32768)
[    0.568587] MPTCP token hash table entries: 4096 (order: 4, 98304 bytes, linear)
[    0.568700] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.568883] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.569115] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.569234] NET: Registered PF_XDP protocol family
[    0.569348] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.569419] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.569527] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.569634] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
[    0.569800] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    0.569900] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    0.570381] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.571421] PCI: CLS 0 bytes, default 64
[    0.571529] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.571630] software IO TLB: mapped [mem 0x00000000ab000000-0x00000000af000000] (64MB)
[    0.575326] Trying to unpack rootfs image as initramfs...
[    0.582878] ACPI: bus type thunderbolt registered
[    0.583153] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x25632a14f90, max_idle_ns: 440795228339 ns
[    0.632780] Initialise system trusted keyrings
[    0.632912] Key type blacklist registered
[    0.636809] workingset: timestamp_bits=36 max_order=20 bucket_order=0
[    0.636933] zbud: loaded
[    0.639497] integrity: Platform Keyring initialized
[    0.639582] integrity: Machine keyring initialized
[    0.649332] NET: Registered PF_ALG protocol family
[    0.649462] xor: automatically using best checksumming function   avx       
[    0.649570] Key type asymmetric registered
[    0.649679] Asymmetric key parser 'x509' registered
[    0.649784] Running certificate verification selftests
[    0.653395] Loaded X.509 cert 'Certificate verification self-testing key: f58703bb33ce1b73ee02eccdee5b8817518fe3db'
[    0.654253] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    0.654463] io scheduler mq-deadline registered
[    0.654534] io scheduler kyber registered
[    0.654663] io scheduler bfq registered
[    0.659961] atomic64_test: passed for x86-64 platform with CX8 and with SSE
[    0.660340] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.660593] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    0.663966] ACPI: button: Power Button [PWRF]
[    0.664641] ACPI: \_SB_.LNKC: Enabled at IRQ 10
[    0.664830] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    0.665328] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
[    0.666428] ACPI: \_SB_.LNKA: Enabled at IRQ 10
[    0.666566] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
[    0.667559] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.668573] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    0.670746] Non-volatile memory driver v1.3
[    0.675579] rdac: device handler registered
[    0.675687] hp_sw: device handler registered
[    0.675781] emc: device handler registered
[    0.675932] alua: device handler registered
[    0.677364] uhci_hcd 0000:00:01.2: UHCI Host Controller
[    0.677502] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
[    0.677630] uhci_hcd 0000:00:01.2: detected 2 ports
[    0.677866] uhci_hcd 0000:00:01.2: irq 11, io port 0x0000c040
[    0.678178] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.14
[    0.678277] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.678347] usb usb1: Product: UHCI Host Controller
[    0.678454] usb usb1: Manufacturer: Linux 5.14.0-503.40.1.el9_5.x86_64 uhci_hcd
[    0.678560] usb usb1: SerialNumber: 0000:00:01.2
[    0.678811] hub 1-0:1.0: USB hub found
[    0.678919] hub 1-0:1.0: 2 ports detected
[    0.679385] usbcore: registered new interface driver usbserial_generic
[    0.679498] usbserial: USB Serial support registered for generic
[    0.679693] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    0.680953] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.681075] serio: i8042 AUX port at 0x60,0x64 irq 12
[    0.681359] mousedev: PS/2 mouse device common for all mice
[    0.681628] rtc_cmos 00:00: RTC can wake from S4
[    0.682918] rtc_cmos 00:00: registered as rtc0
[    0.683241] rtc_cmos 00:00: setting system clock to 2025-12-04T19:53:54 UTC (1764878034)
[    0.683699] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
[    0.683831] intel_pstate: CPU model not supported
[    0.684436] hid: raw HID events driver (C) Jiri Kosina
[    0.684625] usbcore: registered new interface driver usbhid
[    0.684721] usbhid: USB HID core driver
[    0.684838] drop_monitor: Initializing network drop monitor service
[    0.690616] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[    0.691749] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
[    0.692339] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
[    0.707239] Initializing XFRM netlink socket
[    0.707395] NET: Registered PF_INET6 protocol family
[    1.222996] Freeing initrd memory: 30656K
[    1.234094] Segment Routing with IPv6
[    1.234245] NET: Registered PF_PACKET protocol family
[    1.234438] mpls_gso: MPLS GSO support
[    1.236888] IPI shorthand broadcast: enabled
[    1.237011] AVX version of gcm_enc/dec engaged.
[    1.238669] AES CTR mode by8 optimization enabled
[    1.238860] sched_clock: Marking stable (1080005805, 158253566)->(1295232154, -56972783)
[    1.241236] registered taskstats version 1
[    1.242336] Loading compiled-in X.509 certificates
[    1.244115] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: db7d4e4172eaee37bbfd93a756e5dfd7b27c4494'
[    1.244706] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80'
[    1.245278] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8'
[    1.245612] Loaded X.509 cert 'RH-IMA-CA: Red Hat IMA CA: fb31825dd0e073685b264e3038963673f753959a'
[    1.245729] Loaded X.509 cert 'Nvidia GPU OOT signing 001: 55e1cef88193e60419f0b0ec379c49f77545acf0'
[    1.249809] page_owner is disabled
[    1.249967] Key type big_key registered
[    1.259138] Key type encrypted registered
[    1.259298] ima: No TPM chip found, activating TPM-bypass!
[    1.259372] Loading compiled-in module X.509 certificates
[    1.259964] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: db7d4e4172eaee37bbfd93a756e5dfd7b27c4494'
[    1.260046] ima: Allocated hash algorithm: sha256
[    1.260135] ima: No architecture policies found
[    1.260276] evm: Initialising EVM extended attributes:
[    1.260376] evm: security.selinux
[    1.260481] evm: security.SMACK64 (disabled)
[    1.260587] evm: security.SMACK64EXEC (disabled)
[    1.260694] evm: security.SMACK64TRANSMUTE (disabled)
[    1.260763] evm: security.SMACK64MMAP (disabled)
[    1.260868] evm: security.apparmor (disabled)
[    1.260947] evm: security.ima
[    1.261048] evm: security.capability
[    1.261117] evm: HMAC attrs: 0x1
[    1.476840] PM: hibernation: Checking hibernation image partition UUID=dca934f6-a33b-47f1-8763-afe07d62039b
[    1.477132] clk: Disabling unused clocks
[    1.480344] Freeing unused decrypted memory: 2028K
[    1.483443] Freeing unused kernel image (initmem) memory: 3976K
[    1.484618] Write protecting the kernel read-only data: 30720k
[    1.487310] Freeing unused kernel image (rodata/data gap) memory: 1424K
[    1.533052] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.533239] x86/mm: Checking user space page tables
[    1.576148] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.576342] Run /init as init process
[    1.576454]   with arguments:
[    1.576456]     /init
[    1.576457]   with environment:
[    1.576459]     HOME=/
[    1.576460]     TERM=linux
[    1.576461]     BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-5.14.0-503.40.1.el9_5.x86_64
[    1.589272] systemd[1]: systemd 252-46.el9_5.2 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    1.589606] systemd[1]: Detected virtualization kvm.
[    1.589720] systemd[1]: Detected architecture x86-64.
[    1.589826] systemd[1]: Running in initrd.
[    1.599238] systemd[1]: Hostname set to <trevis-68vm5.trevis.whamcloud.com>.
[    1.706735] systemd[1]: Queued start job for default target Initrd Default Target.
[    1.719083] systemd[1]: Created slice Slice /system/systemd-hibernate-resume.
[    1.721827] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    1.724617] systemd[1]: Reached target Initrd /usr File System.
[    1.726777] systemd[1]: Reached target Path Units.
[    1.728598] systemd[1]: Reached target Slice Units.
[    1.730422] systemd[1]: Reached target Swaps.
[    1.732082] systemd[1]: Reached target Timer Units.
[    1.734153] systemd[1]: Listening on D-Bus System Message Bus Socket.
[    1.736640] systemd[1]: Listening on Journal Socket (/dev/log).
[    1.738930] systemd[1]: Listening on Journal Socket.
[    1.740980] systemd[1]: Listening on udev Control Socket.
[    1.743141] systemd[1]: Listening on udev Kernel Socket.
[    1.745110] systemd[1]: Reached target Socket Units.
[    1.757589] systemd[1]: Starting Create List of Static Device Nodes...
[    1.762290] systemd[1]: Starting Journal Service...
[    1.764334] systemd[1]: Load Kernel Modules was skipped because no trigger condition checks were met.
[    1.767549] systemd[1]: Starting Apply Kernel Variables...
[    1.771455] systemd[1]: Starting Create System Users...
[    1.774707] systemd[1]: Starting Setup Virtual Console...
[    1.778352] systemd[1]: Finished Create List of Static Device Nodes.
[    1.800851] systemd[1]: Finished Apply Kernel Variables.
[    1.803891] systemd[1]: Finished Create System Users.
[    1.808541] systemd[1]: Starting Create Static Device Nodes in /dev...
[    1.822131] systemd[1]: Finished Create Static Device Nodes in /dev.
[    1.826118] systemd[1]: Started Journal Service.
[    2.251208] virtio_blk virtio1: 1/0/0 default/read/poll queues
[    2.255694] virtio_blk virtio1: [vda] 209715200 512-byte logical blocks (107 GB/100 GiB)
[    2.258373]  vda: vda1 vda2
[    2.316762] PM: hibernation: Configured hibernation resume from disk to 265289730
[    2.316779] PM: hibernation: Hibernation image partition 253:2 present
[    2.316790] PM: hibernation: Looking for hibernation image.
[    2.318447] PM: Image not found (code -22)
[    2.318459] PM: hibernation: Hibernation image not present or could not be loaded.
[    2.345160] libata version 3.00 loaded.
[    2.348716] ata_piix 0000:00:01.1: version 2.13
[    2.350806] scsi host0: ata_piix
[    2.352045] scsi host1: ata_piix
[    2.352918] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14 lpm-pol 0
[    2.353006] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15 lpm-pol 0
[    2.914016] EXT4-fs (vda1): mounted filesystem 83f639ed-9bac-410b-92f1-8e2ef103e35e ro with ordered data mode. Quota mode: none.
[    3.136586] systemd-journald[229]: Received SIGTERM from PID 1 (systemd).
[    3.186710] systemd[1]: systemd 252-46.el9_5.2 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    3.187020] systemd[1]: Detected virtualization kvm.
[    3.187116] systemd[1]: Detected architecture x86-64.
[    3.241828] systemd-fstab-generator[430]: Checking was requested for "10.240.32.204:/export/autotest", but it is not a device.
[    3.244367] systemd-fstab-generator[430]: Checking was requested for "10.240.32.204:/export/scratch", but it is not a device.
[    3.279345] systemd-rc-local-generator[435]: /etc/rc.d/rc.local is not marked executable, skipping.
[    3.365297] systemd[1]: /usr/lib/systemd/system/munge.service:10: PIDFile= references a path below legacy directory /var/run/, updating /var/run/munge/munged.pid \xe2\x86\x92 /run/munge/munged.pid; please update the unit file accordingly.
[    3.435611] systemd[1]: initrd-switch-root.service: Deactivated successfully.
[    3.435985] systemd[1]: Stopped Switch Root.
[    3.438938] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[    3.439630] systemd[1]: Created slice Slice /system/getty.
[    3.443005] systemd[1]: Created slice Slice /system/modprobe.
[    3.445665] systemd[1]: Created slice Slice /system/serial-getty.
[    3.448328] systemd[1]: Created slice Slice /system/sshd-keygen.
[    3.450810] systemd[1]: Created slice User and Session Slice.
[    3.452956] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    3.455628] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    3.458312] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    3.461302] systemd[1]: Reached target Local Encrypted Volumes.
[    3.463334] systemd[1]: Stopped target Switch Root.
[    3.465024] systemd[1]: Stopped target Initrd File Systems.
[    3.466942] systemd[1]: Stopped target Initrd Root File System.
[    3.468948] systemd[1]: Reached target Local Integrity Protected Volumes.
[    3.471208] systemd[1]: Reached target Host and Network Name Lookups.
[    3.473342] systemd[1]: Reached target Path Units.
[    3.475004] systemd[1]: Reached target Slice Units.
[    3.476683] systemd[1]: Reached target Local Verity Protected Volumes.
[    3.478986] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[    3.481344] systemd[1]: Listening on LVM2 poll daemon socket.
[    3.493371] systemd[1]: Listening on RPCbind Server Activation Socket.
[    3.495657] systemd[1]: Reached target RPC Port Mapper.
[    3.498966] systemd[1]: Listening on Process Core Dump Socket.
[    3.501084] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    3.503468] systemd[1]: Listening on udev Control Socket.
[    3.505468] systemd[1]: Listening on udev Kernel Socket.
[    3.515402] systemd[1]: Mounting Huge Pages File System...
[    3.517926] systemd[1]: Mounting POSIX Message Queue File System...
[    3.523478] systemd[1]: Mounting Kernel Debug File System...
[    3.527556] systemd[1]: Mounting Kernel Trace File System...
[    3.529527] systemd[1]: Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
[    3.532551] systemd[1]: Starting Create List of Static Device Nodes...
[    3.538012] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[    3.544496] systemd[1]: Starting Load Kernel Module configfs...
[    3.548041] systemd[1]: Starting Load Kernel Module drm...
[    3.552382] systemd[1]: Starting Load Kernel Module fuse...
[    3.556683] systemd[1]: Starting Read and set NIS domainname from /etc/sysconfig/network...
[    3.560216] systemd[1]: Stopped Journal Service.
[    3.571567] systemd[1]: Starting Journal Service...
[    3.574033] systemd[1]: Load Kernel Modules was skipped because no trigger condition checks were met.
[    3.577446] systemd[1]: Starting Generate network units from Kernel command line...
[    3.580297] systemd[1]: TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f).
[    3.587470] systemd[1]: Starting Remount Root and Kernel File Systems...
[    3.590102] systemd[1]: Repartition Root Disk was skipped because no trigger condition checks were met.
[    3.593695] systemd[1]: Starting Apply Kernel Variables...
[    3.597340] systemd[1]: Starting Coldplug All udev Devices...
[    3.602343] systemd[1]: Mounted Huge Pages File System.
[    3.604506] systemd[1]: Mounted POSIX Message Queue File System.
[    3.606810] systemd[1]: Mounted Kernel Debug File System.
[    3.609075] systemd[1]: Mounted Kernel Trace File System.
[    3.611468] systemd[1]: Finished Create List of Static Device Nodes.
[    3.614448] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[    3.616174] systemd[1]: Finished Load Kernel Module configfs.
[    3.619133] systemd[1]: Finished Read and set NIS domainname from /etc/sysconfig/network.
[    3.622473] systemd[1]: Finished Generate network units from Kernel command line.
[    3.626074] systemd[1]: Reached target Preparation for Network.
[    3.631417] systemd[1]: Mounting Kernel Configuration File System...
[    3.636367] systemd[1]: Finished Apply Kernel Variables.
[    3.639121] systemd[1]: Mounted Kernel Configuration File System.
[    3.726297] EXT4-fs (vda1): re-mounted 83f639ed-9bac-410b-92f1-8e2ef103e35e r/w. Quota mode: none.
[    3.728563] systemd[1]: Finished Remount Root and Kernel File Systems.
[    3.730950] systemd[1]: First Boot Wizard was skipped because of an unmet condition check (ConditionFirstBoot=yes).
[    3.731678] systemd[1]: Rebuild Hardware Database was skipped because of an unmet condition check (ConditionNeedsUpdate=/etc).
[    3.737505] systemd[1]: Starting Load/Save OS Random Seed...
[    3.739515] systemd[1]: Create System Users was skipped because no trigger condition checks were met.
[    3.741162] systemd[1]: Starting Create Static Device Nodes in /dev...
[    3.759142] systemd[1]: Started Journal Service.
[    3.788406] systemd-journald[458]: Received client request to flush runtime journal.
[    3.822277] random: crng init done
[    3.824097] fuse: init (API version 7.36)
[    3.860541] ACPI: bus type drm_connector registered
[    4.068633] Adding 2846716k swap on /dev/vda2.  Priority:-2 extents:1 across:2846716k 
[    4.148658] input: PC Speaker as /devices/platform/pcspkr/input/input5
[    4.165872] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0
[    4.800432] RPC: Registered named UNIX socket transport module.
[    4.800526] RPC: Registered udp transport module.
[    4.800596] RPC: Registered tcp transport module.
[    4.800659] RPC: Registered tcp-with-tls transport module.
[    4.800723] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.290886] FS-Cache: Loaded
[    5.406928] Key type dns_resolver registered
[    5.683284] NFS: Registering the id_resolver key type
[    5.684555] Key type id_resolver registered
[    5.685246] Key type id_legacy registered
[   48.584877] Autotest: Nodes configured in 1 minutes (lustre-master_full-part-1_4681.175)
[   48.842964] Autotest: Node provisioning/configuration took 20 minutes (lustre-master_full-part-1_4681.175)
[   49.335358] Autotest: Starting test framework on controller trevis-42vm5 (lustre-master_full-part-1_4681.175)
[   54.516157] Autotest: Test framework started (lustre-master_full-part-1_4681.175)
[   55.101052] Autotest: Test framework pid=1251 (lustre-master_full-part-1_4681.175)
[   55.387791] Autotest: Test running for 0 minutes (lustre-master_full-part-1_4681.175)
[  205.230103] libcfs: loading out-of-tree module taints kernel.
[  205.230201] libcfs: module verification failed: signature and/or required key missing - tainting kernel
[  205.250034] Key type ._llcrypt registered
[  205.250161] Key type .llcrypt registered
[  205.492511] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
[  205.496261] alg: No test for adler32 (adler32-zlib)
[  206.398115] Lustre: Lustre: Build Version: 2.17.0_RC1
[  206.465034] LNet: Added LNI 10.240.41.196@tcp [8/256/0/180]
[  207.577716] Lustre: Mounted lustre-client
[  207.762274] Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
[  208.171956] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
[  210.004319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  210.004534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  210.237398] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  210.240549] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  212.251009] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[  212.472774] Lustre: DEBUG MARKER: Using TIMEOUT=20
[  212.740574] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[  221.050718] Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
[  221.404474] Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
[  228.380592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Client: 2.17.0.RC1
[  228.602638] Lustre: DEBUG MARKER: Client: 2.17.0.RC1
[  229.203526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MDS: 2.15.8
[  229.436882] Lustre: DEBUG MARKER: MDS: 2.15.8
[  230.061896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OSS: 2.15.8
[  230.284939] Lustre: DEBUG MARKER: OSS: 2.15.8
[  230.570948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: runtests ============----- Thu Dec  4 07:57:44 PM UTC 2025
[  230.789476] Lustre: DEBUG MARKER: -----============= acceptance-small: runtests ============----- Thu Dec 4 07:57:44 PM UTC 2025
[  236.418499] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 
[  236.639172] Lustre: DEBUG MARKER: excepting tests:
[  236.922315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === runtests: start setup 19:57:50 \(1764878270\) ===
[  237.138988] Lustre: DEBUG MARKER: === runtests: start setup 19:57:50 (1764878270) ===
[  237.365646] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  238.884213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[  238.908873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[  238.950597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[  239.169758] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[  239.216933] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[  239.283849] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[  242.881828] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[  243.278248] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[  250.801890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[  251.026733] Lustre: DEBUG MARKER: Using TIMEOUT=20
[  251.305667] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[  253.899733] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  255.433429] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  255.435778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  255.708301] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  255.717897] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  256.137197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  256.193320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  256.425687] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  256.463940] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  257.792015] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === runtests: finish setup 19:58:11 \(1764878291\) ===
[  258.020026] Lustre: DEBUG MARKER: === runtests: finish setup 19:58:11 (1764878291) ===
[  269.811192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test 1: All Runtests ========================= 19:58:23 \(1764878303\)
[  270.038978] Lustre: DEBUG MARKER: == runtests test 1: All Runtests ========================= 19:58:23 (1764878303)
[  275.396988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark touching \/mnt\/lustre at Thu Dec  4 07:58:29 PM UTC 2025 \(@1764878309\)
[  275.621516] Lustre: DEBUG MARKER: touching /mnt/lustre at Thu Dec 4 07:58:29 PM UTC 2025 (@1764878309)
[  275.893696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create an empty file \/mnt\/lustre\/hosts.11209
[  276.114290] Lustre: DEBUG MARKER: create an empty file /mnt/lustre/hosts.11209
[  276.399397] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.11209
[  276.622618] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.11209
[  276.901997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing \/etc\/hosts and \/mnt\/lustre\/hosts.11209
[  277.121366] Lustre: DEBUG MARKER: comparing /etc/hosts and /mnt/lustre/hosts.11209
[  277.445720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark renaming \/mnt\/lustre\/hosts.11209 to \/mnt\/lustre\/hosts.11209.ren
[  277.660527] Lustre: DEBUG MARKER: renaming /mnt/lustre/hosts.11209 to /mnt/lustre/hosts.11209.ren
[  277.937411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.11209 again
[  278.157419] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.11209 again
[  278.436038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.11209
[  278.653645] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.11209
[  278.927943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/hosts.11209
[  279.145436] Lustre: DEBUG MARKER: removing /mnt/lustre/hosts.11209
[  279.418550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.11209.2
[  279.637124] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.11209.2
[  279.913702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.11209.2 to 123 bytes
[  280.133192] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.11209.2 to 123 bytes
[  280.408597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating \/mnt\/lustre\/d1.runtests
[  280.627673] Lustre: DEBUG MARKER: creating /mnt/lustre/d1.runtests
[  284.933036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying 1659 files from \/etc, \/usr\/bin to  \/mnt\/lustre\/d1.runtests\/etc, \/mnt\/lustre\/d1.runtests\/usr\/bin at Thu Dec  4 07:58:38 PM UTC 2025
[  285.167104] Lustre: DEBUG MARKER: copying 1659 files from /etc, /usr/bin to /mnt/lustre/d1.runtests/etc, /mnt/lustre/d1.runtests/usr/bin at Thu Dec 4 07:58:38 PM UTC 2025
[  302.816747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing 1659 newly copied files at Thu Dec  4 07:58:56 PM UTC 2025
[  303.050417] Lustre: DEBUG MARKER: comparing 1659 newly copied files at Thu Dec 4 07:58:56 PM UTC 2025
[  308.933631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark running createmany -d \/mnt\/lustre\/d1.runtests\/d 1659
[  309.161181] Lustre: DEBUG MARKER: running createmany -d /mnt/lustre/d1.runtests/d 1659
[  309.706479] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=ha
[  310.841230] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=super+ioctl+neterror+warning+dlmtrace+error+emerg+ha+rpctrace+vfstrace+config+console+lfsck
[  311.733587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark finished at Thu Dec  4 07:59:05 PM UTC 2025 \(36\)
[  311.951928] Lustre: DEBUG MARKER: finished at Thu Dec 4 07:59:05 PM UTC 2025 (36)
[  312.144336] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre opts:;
lsof /mnt/lustre || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre | sort -u);
	if [ -n "
[  312.530006] LustreError: 10666:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91fe98960800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[  312.530718] LustreError: 10666:0:(obd_class.h:478:obd_check_dev()) Device 10 not setup
[  312.549171] Lustre: Unmounted lustre-client
[  335.026721] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[  358.354516] Autotest: Test running for 5 minutes (lustre-master_full-part-1_4681.175)
[  363.862879] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[  364.290708] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[  369.835546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  370.057397] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  377.089243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  377.325200] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  384.072952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  384.300344] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  391.006468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  391.233401] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  398.065829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  398.293505] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  405.142258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  405.356676] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  412.330894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  412.553628] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  419.539471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  419.763633] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  420.894349] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
[  421.315599] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:
[  421.543317] Lustre: Mounted lustre-client
[  421.735991] Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
[  422.153638] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  423.678818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  423.680648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  423.918601] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  423.918977] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  425.963755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[  426.193811] Lustre: DEBUG MARKER: Using TIMEOUT=20
[  426.472205] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[  430.430347] Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
[  430.796591] Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
[  436.269340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark After first remount. comparing 1659 previously copied files
[  436.525126] Lustre: DEBUG MARKER: After first remount. comparing 1659 previously copied files
[  448.351947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark running statmany -s \/mnt\/lustre\/d1.runtests\/d 1659 3318
[  448.584880] Lustre: DEBUG MARKER: running statmany -s /mnt/lustre/d1.runtests/d 1659 3318
[  449.860887] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre opts:;
lsof /mnt/lustre || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre | sort -u);
	if [ -n "
[  450.237709] LustreError: 12894:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91ff82438000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[  450.237974] LustreError: 12894:0:(lov_obd.c:780:lov_cleanup()) Skipped 6 previous similar messages
[  450.238765] LustreError: 12894:0:(obd_class.h:478:obd_check_dev()) Device 10 not setup
[  450.238888] LustreError: 12894:0:(obd_class.h:478:obd_check_dev()) Skipped 17 previous similar messages
[  450.255035] Lustre: Unmounted lustre-client
[  467.708494] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[  495.240738] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[  495.667985] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[  501.135519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  501.374154] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  508.355337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  508.580595] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  515.320261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  515.547090] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  522.250448] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  522.476358] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  529.414666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  529.664389] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  536.288150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  536.517200] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  543.411421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  543.666545] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  550.596879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  550.828808] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  551.856005] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
[  552.286163] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:
[  552.526283] Lustre: Mounted lustre-client
[  552.725249] Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
[  553.161455] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  554.697870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  554.709305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  554.957479] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  554.957479] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[  557.004763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[  557.237365] Lustre: DEBUG MARKER: Using TIMEOUT=20
[  557.516673] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[  565.245665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark After second remount. running unlinkmany -d \/mnt\/lustre\/d1.runtests\/d 1659
[  565.487391] Lustre: DEBUG MARKER: After second remount. running unlinkmany -d /mnt/lustre/d1.runtests/d 1659
[  566.152980] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
[  568.105500] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=super+ioctl+neterror+warning+dlmtrace+error+emerg+ha+rpctrace+vfstrace+config+console+lfsck
[  569.136691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/d1.runtests
[  569.363695] Lustre: DEBUG MARKER: removing /mnt/lustre/d1.runtests
[  572.141320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark renaming \/mnt\/lustre\/hosts.11209.ren to \/mnt\/lustre\/hosts.11209
[  572.402094] Lustre: DEBUG MARKER: renaming /mnt/lustre/hosts.11209.ren to /mnt/lustre/hosts.11209
[  572.727273] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.11209
[  572.971057] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.11209
[  573.273470] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/hosts.11209
[  573.506766] Lustre: DEBUG MARKER: removing /mnt/lustre/hosts.11209
[  573.827061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark verifying \/mnt\/lustre\/hosts.11209.2 is 123 bytes
[  574.063872] Lustre: DEBUG MARKER: verifying /mnt/lustre/hosts.11209.2 is 123 bytes
[  574.388400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done
[  574.626110] Lustre: DEBUG MARKER: done
[  587.225820] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Space was freed: now 8932kB, was 8872kB.
[  587.481497] Lustre: DEBUG MARKER: Space was freed: now 8932kB, was 8872kB.
[  588.789265] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test complete, duration 358 sec ============== 20:03:42 \(1764878622\)
[  589.020578] Lustre: DEBUG MARKER: == runtests test complete, duration 358 sec ============== 20:03:42 (1764878622)
[  589.326791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === runtests: start cleanup 20:03:43 \(1764878623\) ===
[  589.560385] Lustre: DEBUG MARKER: === runtests: start cleanup 20:03:43 (1764878623) ===
[  590.969530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === runtests: finish cleanup 20:03:44 \(1764878624\) ===
[  591.200854] Lustre: DEBUG MARKER: === runtests: finish cleanup 20:03:44 (1764878624) ===
[  591.518850] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: replay-dual ============----- Thu Dec  4 08:03:45 PM UTC 2025
[  591.753066] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-dual ============----- Thu Dec 4 08:03:45 PM UTC 2025
[  597.560895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 14b 21b
[  597.802535] Lustre: DEBUG MARKER: excepting tests: 14b 21b
[  598.117595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === replay-dual: start setup 20:03:51 \(1764878631\) ===
[  598.347951] Lustre: DEBUG MARKER: === replay-dual: start setup 20:03:51 (1764878631) ===
[  598.594152] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre2
[  599.059969] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre2' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre2
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tc
[  599.282204] Lustre: Mounted lustre-client
[  599.476779] Lustre: DEBUG MARKER: mount | grep /mnt/lustre2' '
[  599.901807] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  601.449320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  601.472127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  601.711000] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  601.711007] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  601.921100] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  603.476176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[  603.482944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[  603.588806] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[  603.751304] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[  603.872304] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[  603.886674] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[  607.582195] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[  608.048601] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[  615.596835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[  615.852590] Lustre: DEBUG MARKER: Using TIMEOUT=20
[  616.148530] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[  618.949066] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  620.668413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  620.669853] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  620.966864] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  620.975250] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  621.198038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  621.274715] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  621.468262] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  621.557669] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[  622.913012] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === replay-dual: finish setup 20:04:16 \(1764878656\) ===
[  623.147322] Lustre: DEBUG MARKER: === replay-dual: finish setup 20:04:16 (1764878656) ===
[  634.999017] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 0a: expired recovery with lost client ========================================================== 20:04:28 \(1764878668\)
[  635.230930] Lustre: DEBUG MARKER: == replay-dual test 0a: expired recovery with lost client ========================================================== 20:04:28 (1764878668)
[  636.021339] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[  636.451036] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[  640.625714] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[  640.625843] Lustre: Skipped 1 previous similar message
[  657.008095] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764878674/real 1764878674]  req@ffff91fe97d3d380 x1850608965448192/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764878690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[  657.008361] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[  658.647099] Autotest: Test running for 10 minutes (lustre-master_full-part-1_4681.175)
[  663.263983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  663.512559] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  667.248878] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a0f395b to 0xbe06701d5a123e10
[  667.249959] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[  739.453770] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[  752.552426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 0b: lost client during waiting for next transno ========================================================== 20:06:26 \(1764878786\)
[  752.782473] Lustre: DEBUG MARKER: == replay-dual test 0b: lost client during waiting for next transno ========================================================== 20:06:26 (1764878786)
[  753.393798] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[  753.805216] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[  757.358824] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[  777.835266] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[  777.838829] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a123e10 to 0xbe06701d5a1257bd
[  777.841589] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[  777.841665] Lustre: Skipped 1 previous similar message
[  780.322547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  780.542320] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  856.451879] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[  859.601434] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 1: \|X\| simple create ================= 20:08:13 \(1764878893\)
[  859.834590] Lustre: DEBUG MARKER: == replay-dual test 1: |X| simple create ================= 20:08:13 (1764878893)
[  860.534784] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[  860.947696] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[  868.967221] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[  868.967332] Lustre: Skipped 2 previous similar messages
[  884.325951] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764878902/real 1764878902]  req@ffff91fe97d3da00 x1850608965476096/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764878918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[  884.326179] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[  884.328115] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a1257bd to 0xbe06701d5a125d97
[  884.328941] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[  884.329049] Lustre: Skipped 1 previous similar message
[  886.797359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  887.020502] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  889.738578] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  891.287870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  891.445189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  891.476847] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  891.571969] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  891.738950] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  891.749599] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  891.775073] Lustre: DEBUG MARKER: lctl get_param -n at_max
[  891.929619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  892.193937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  892.206301] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  892.247867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  892.447414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  892.448828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  894.928544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 2: \|X\| mkdir adir ==================== 20:08:48 \(1764878928\)
[  895.158314] Lustre: DEBUG MARKER: == replay-dual test 2: |X| mkdir adir ==================== 20:08:48 (1764878928)
[  895.727367] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[  896.152497] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[  899.686047] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[  915.044210] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764878933/real 1764878933]  req@ffff91fe9ba33400 x1850608965486720/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764878949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[  915.044371] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[  922.108217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  922.330197] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  925.284849] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a125d97 to 0xbe06701d5a1264c1
[  925.287740] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[  925.287867] Lustre: Skipped 2 previous similar messages
[  929.613517] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  931.215582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  931.253739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  931.260825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  931.503180] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  931.558875] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  931.596760] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  931.617780] Lustre: DEBUG MARKER: lctl get_param -n at_max
[  931.842897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  931.981070] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  932.011266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  932.193370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  932.287371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  932.313555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  934.794710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 3: \|X\| mkdir adir, mkdir adir\/bdir === 20:09:28 \(1764878968\)
[  935.037597] Lustre: DEBUG MARKER: == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 20:09:28 (1764878968)
[  935.644532] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[  936.063439] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[  939.620391] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[  939.620566] Lustre: Skipped 1 previous similar message
[  954.978370] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764878973/real 1764878973]  req@ffff91fe9ba32a40 x1850608965501312/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764878989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[  954.978597] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[  959.575514] Autotest: Test running for 15 minutes (lustre-master_full-part-1_4681.175)
[  962.061741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  962.270612] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[  965.218907] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a1264c1 to 0xbe06701d5a126aef
[  965.221855] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[  965.221967] Lustre: Skipped 2 previous similar messages
[  971.123873] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[  972.656760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  972.709022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  972.720184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[  972.983136] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  973.034068] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  973.041948] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[  973.062573] Lustre: DEBUG MARKER: lctl get_param -n at_max
[  973.419569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  973.424029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  973.450551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[  973.713848] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  973.762939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  973.764265] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[  976.256239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 4: \|X\| mkdir adir \(-EEXIST\), mkdir adir\/bdir ========================================================== 20:10:10 \(1764879010\)
[  976.494991] Lustre: DEBUG MARKER: == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 20:10:10 (1764879010)
[  977.143046] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[  977.560920] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[  984.674173] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[  984.674291] Lustre: Skipped 1 previous similar message
[ 1000.928222] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764879018/real 1764879018]  req@ffff91fe97d3e700 x1850608965517312/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764879034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1000.928454] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 1000.937537] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a126aef to 0xbe06701d5a126ff0
[ 1000.940283] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[ 1000.940359] Lustre: Skipped 2 previous similar messages
[ 1003.654203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1003.916445] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1012.027393] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1013.576819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1013.654740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1013.722020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1013.857872] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1013.984599] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1014.022136] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1014.042326] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1014.200719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1014.398525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1014.432389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1014.506748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1014.689960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1014.690256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1017.195537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 5: open, unlink \|X\| close ============ 20:10:50 \(1764879050\)
[ 1017.426658] Lustre: DEBUG MARKER: == replay-dual test 5: open, unlink |X| close ============ 20:10:50 (1764879050)
[ 1018.054633] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1018.493963] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1024.608240] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1024.608447] Lustre: Skipped 1 previous similar message
[ 1040.990441] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764879058/real 1764879058]  req@ffff91fe97d3e080 x1850608965532288/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764879074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1040.990617] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 1044.496496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1044.719208] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1051.215045] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a126ff0 to 0xbe06701d5a127990
[ 1053.015029] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1054.561474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1054.646443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1054.664988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1054.818617] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1054.955690] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1054.973567] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1055.000810] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1055.139266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1055.356658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1055.392637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1055.503166] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1055.655024] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1055.671895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1058.134686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 6: open1, open2, unlink \|X\| close1 [fail mds1] close2 ========================================================== 20:11:31 \(1764879091\)
[ 1058.355096] Lustre: DEBUG MARKER: == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 20:11:31 (1764879091)
[ 1058.966720] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1059.375050] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1066.590338] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1066.590482] Lustre: Skipped 1 previous similar message
[ 1082.972498] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764879100/real 1764879100]  req@ffff91fe97d3d6c0 x1850608965547776/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764879116 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1082.972767] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 1082.983048] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[ 1082.983127] Lustre: Skipped 5 previous similar messages
[ 1085.396416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1085.618135] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1093.982994] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1095.525154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1095.706305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1095.738359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1095.864944] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1096.024428] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1096.034072] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1096.064008] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1096.209762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1096.445623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1096.484413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1096.562309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1096.751686] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1096.753523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1099.266314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 8: replay of resent request ========== 20:12:13 \(1764879133\)
[ 1099.502800] Lustre: DEBUG MARKER: == replay-dual test 8: replay of resent request ========== 20:12:13 (1764879133)
[ 1100.124593] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1100.557897] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1137.242020] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764879155/real 1764879155]  req@ffff91fe97d3ed80 x1850608965566336/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764879171 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1142.734382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1142.949346] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1145.645813] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1147.198348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1147.208837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1147.273002] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1147.483236] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a127f16 to 0xbe06701d5a1285d7
[ 1147.483341] Lustre: Skipped 1 previous similar message
[ 1147.519604] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1147.576110] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1147.590954] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1147.611763] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1147.890034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1147.970349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1147.990994] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1148.169848] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1148.287710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1148.295573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1150.762505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 9: resending a replayed create ======= 20:13:04 \(1764879184\)
[ 1150.987341] Lustre: DEBUG MARKER: == replay-dual test 9: resending a replayed create ======= 20:13:04 (1764879184)
[ 1151.556270] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1151.996697] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1157.722328] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1157.722505] Lustre: Skipped 3 previous similar messages
[ 1173.848394] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 1173.848520] LustreError: Skipped 1 previous similar message
[ 1178.212102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1178.448887] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1201.969651] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1203.538748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1203.565456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1203.598300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1203.879784] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1203.908486] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1203.926807] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1203.933224] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1204.268707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1204.308615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1204.325796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1204.608894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1204.622018] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1204.650978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1207.426653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 10: resending a replayed unlink ====== 20:14:01 \(1764879241\)
[ 1207.658826] Lustre: DEBUG MARKER: == replay-dual test 10: resending a replayed unlink ====== 20:14:01 (1764879241)
[ 1208.316939] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1208.726353] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1230.421749] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764879248/real 1764879248]  req@ffff91fe9ba323c0 x1850608965595392/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764879264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1230.421881] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1234.875706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1235.086788] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1240.662309] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a128bc6 to 0xbe06701d5a1292e9
[ 1240.662423] Lustre: Skipped 1 previous similar message
[ 1240.667597] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[ 1240.667708] Lustre: Skipped 8 previous similar messages
[ 1253.165533] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1254.677708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1254.802467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1254.848874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1254.981939] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1255.108719] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1255.140884] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1255.164247] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1255.324520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1255.579695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1255.588086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1255.591745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1255.821777] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1255.823266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1258.633436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 11: both clients timeout during replay ========================================================== 20:14:52 \(1764879292\)
[ 1258.856644] Lustre: DEBUG MARKER: == replay-dual test 11: both clients timeout during replay ========================================================== 20:14:52 (1764879292)
[ 1259.444795] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1259.853785] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1260.773482] Autotest: Test running for 20 minutes (lustre-master_full-part-1_4681.175)
[ 1286.120809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1286.340838] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1287.194114] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1288.739476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1288.778218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1288.797987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1289.057670] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1289.098533] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1289.110026] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1289.130227] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1303.635042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 14 sec
[ 1303.645068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 14 sec
[ 1303.653052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 14 sec
[ 1303.952508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec
[ 1303.960433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec
[ 1303.991217] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec
[ 1305.614564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 12: open resend timeout ============== 20:15:39 \(1764879339\)
[ 1305.837730] Lustre: DEBUG MARKER: == replay-dual test 12: open resend timeout ============== 20:15:39 (1764879339)
[ 1306.421004] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1306.831880] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1313.362720] Lustre: lustre-MDT0000-mdc-ffff91ff828fc000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1313.362887] Lustre: Skipped 6 previous similar messages
[ 1329.745307] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 1329.745445] LustreError: Skipped 2 previous similar messages
[ 1333.283583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1333.506722] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1350.471686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 13: close resend timeout ============= 20:16:24 \(1764879384\)
[ 1350.694128] Lustre: DEBUG MARKER: == replay-dual test 13: close resend timeout ============= 20:16:24 (1764879384)
[ 1351.381478] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1351.788347] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1375.822993] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764879393/real 1764879393]  req@ffff91fe97d3f740 x1850608965630592/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764879409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1375.823095] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 1375.831556] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a12a041 to 0xbe06701d5a12a581
[ 1375.835197] Lustre: Skipped 2 previous similar messages
[ 1378.326026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1378.531650] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1381.955494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b
[ 1382.179807] Lustre: DEBUG MARKER: SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b
[ 1382.475835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 20:16:56 \(1764879416\)
[ 1382.701376] Lustre: DEBUG MARKER: == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 20:16:56 (1764879416)
[ 1383.260060] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1383.668425] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1410.019854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1410.240046] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1480.694028] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1482.288255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1482.324057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1482.346278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1482.604091] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1482.641929] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1482.674281] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1482.695435] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1482.956567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1483.061748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1483.095771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1483.223908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1483.377068] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1483.384835] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1486.815520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 15c: remove multiple OST orphans ===== 20:18:40 \(1764879520\)
[ 1487.053245] Lustre: DEBUG MARKER: == replay-dual test 15c: remove multiple OST orphans ===== 20:18:40 (1764879520)
[ 1487.665307] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1488.081021] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1526.915935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1527.128067] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1534.537799] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[ 1534.537901] Lustre: Skipped 14 previous similar messages
[ 1561.304056] Autotest: Test running for 25 minutes (lustre-master_full-part-1_4681.175)
[ 1595.789230] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1597.373931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1597.390299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1597.421174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1597.685844] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1597.713392] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1597.749977] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1597.770914] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1598.099756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1598.103227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1598.173998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1598.423864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1598.440980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1598.503173] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1601.069973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 16: fail MDS during recovery \(3571\) == 20:20:34 \(1764879634\)
[ 1601.289104] Lustre: DEBUG MARKER: == replay-dual test 16: fail MDS during recovery (3571) == 20:20:34 (1764879634)
[ 1601.912268] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1602.325343] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1606.213701] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1606.213792] Lustre: Skipped 6 previous similar messages
[ 1622.595638] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 1622.595799] LustreError: Skipped 3 previous similar messages
[ 1627.617682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1627.836035] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1632.820089] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a143f63 to 0xbe06701d5a144996
[ 1632.820290] Lustre: Skipped 2 previous similar messages
[ 1649.164531] LustreError: lustre-MDT0000-mdc-ffff91fe98965000: operation mds_reint to node 10.240.38.81@tcp failed: rc = -19
[ 1670.941521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1671.162182] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1749.403338] LustreError: 6139:0:(client.c:3416:ptlrpc_replay_interpret()) @@@ status 0, old was -19  req@ffff91fe97d3e700 x1850608965676416/t81604378626(81604378626) o36->lustre-MDT0000-mdc-ffff91fe98965000@10.240.38.81@tcp:12/10 lens 536/448 e 8 to 0 dl 1764879801 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 1749.699268] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1751.241586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1751.275618] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1751.311223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 1751.550927] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1751.610518] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1751.625418] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1751.645979] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1751.977960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1751.992479] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1752.021437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 1752.282238] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1752.337956] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1752.340569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1755.051186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 17: fail OST during recovery \(3571\) == 20:23:08 \(1764879788\)
[ 1755.278947] Lustre: DEBUG MARKER: == replay-dual test 17: fail OST during recovery (3571) == 20:23:08 (1764879788)
[ 1756.381752] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1756.799959] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 1778.695884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1778.915776] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1816.122731] Lustre: 6139:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764879810/real 1764879810]  req@ffff91fe97d3dd40 x1850608965705728/t0(0) o400->lustre-OST0000-osc-ffff91fe98965000@10.240.41.156@tcp:28/4 lens 224/224 e 2 to 1 dl 1764879850 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295
[ 1816.122961] Lustre: 6139:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 1817.139704] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1817.366236] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 1876.642407] Autotest: Test running for 30 minutes (lustre-master_full-part-1_4681.175)
[ 1884.673051] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 1886.242067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 1886.296916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 1886.324364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 1886.520598] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 1886.595306] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 1886.663233] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 1886.685909] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 1886.863373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
[ 1887.066154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL
IDLE state after 0 sec
[ 1887.082018] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL
IDLE state after 0 sec
[ 1887.160844] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 1887.334711] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL
[ 1887.338067] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL
[ 1910.118223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export \(3822\) ========================================================== 20:25:43 \(1764879943\)
[ 1910.344841] Lustre: DEBUG MARKER: == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 20:25:43 (1764879943)
[ 1955.062717] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 19: resend of open request =========== 20:26:28 \(1764879988\)
[ 1955.290860] Lustre: DEBUG MARKER: == replay-dual test 19: resend of open request =========== 20:26:28 (1764879988)
[ 1955.944323] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 1956.355059] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 2068.951898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2069.181728] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2072.112527] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[ 2072.112636] Lustre: Skipped 6 previous similar messages
[ 2076.644889] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 2078.164682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2078.301406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2078.340455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2078.431682] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2078.609914] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2078.615025] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2078.645389] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 2078.783810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2079.002731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2079.014854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2079.137327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2079.289467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2079.300496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2081.784012] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 20: recovery time is not increasing == 20:28:35 \(1764880115\)
[ 2082.008277] Lustre: DEBUG MARKER: == replay-dual test 20: recovery time is not increasing == 20:28:35 (1764880115)
[ 2082.587092] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 2082.998484] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 2109.459632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2109.681637] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2169.291847] Autotest: Test running for 35 minutes (lustre-master_full-part-1_4681.175)
[ 2251.619168] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 2253.184778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2253.213436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2253.253042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2253.472134] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2253.555805] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2253.580762] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2253.602476] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 2253.812113] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2254.003235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2254.028364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2254.079246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2254.278539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2254.287453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2256.495766] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 2256.912135] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 2264.615081] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2264.615219] Lustre: Skipped 6 previous similar messages
[ 2279.973416] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 2279.973522] LustreError: Skipped 3 previous similar messages
[ 2282.992047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2283.228468] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2290.213964] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a146e2c to 0xbe06701d5a147572
[ 2290.214156] Lustre: Skipped 3 previous similar messages
[ 2423.626051] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 2425.195812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2425.273828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2425.297300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2425.501774] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2425.603105] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2425.610625] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2425.630226] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 2425.856658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2426.001670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2426.058277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2426.113106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2426.305816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2426.305918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2428.894397] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 21a: commit on sharing =============== 20:34:22 \(1764880462\)
[ 2429.135191] Lustre: DEBUG MARKER: == replay-dual test 21a: commit on sharing =============== 20:34:22 (1764880462)
[ 2448.925688] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764880467/real 1764880467]  req@ffff91fe97d3e080 x1850608965787136/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764880483 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2448.925905] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 2454.407953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2454.626785] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2502.307588] Autotest: Test running for 40 minutes (lustre-master_full-part-1_4681.175)
[ 2594.468137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_21b skipping ALWAYS excluded test 21b
[ 2594.706989] Lustre: DEBUG MARKER: SKIP: replay-dual test_21b skipping ALWAYS excluded test 21b
[ 2595.019730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1\/dir ========================================================== 20:37:08 \(1764880628\)
[ 2595.267197] Lustre: DEBUG MARKER: == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply
[ 2595.547000] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_22a needs \>= 2 MDTs
[ 2595.769328] Lustre: DEBUG MARKER: SKIP: replay-dual test_22a needs >= 2 MDTs
[ 2596.088411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0\/M1, c2 mkdir d1\/dir ========================================================== 20:37:09 \(1764880629\)
[ 2596.324252] Lustre: DEBUG MARKER: == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply
[ 2596.615366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_22b needs \>= 2 MDTs
[ 2596.855419] Lustre: DEBUG MARKER: SKIP: replay-dual test_22b needs >= 2 MDTs
[ 2597.180598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1\/dir ========================================================== 20:37:11 \(1764880631\)
[ 2597.415754] Lustre: DEBUG MARKER: == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update
[ 2597.700355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_22c needs \>= 2 MDTs
[ 2597.929748] Lustre: DEBUG MARKER: SKIP: replay-dual test_22c needs >= 2 MDTs
[ 2598.245181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0\/M1,c2 mkdir d1\/dir ========================================================== 20:37:12 \(1764880632\)
[ 2598.472111] Lustre: DEBUG MARKER: == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update
[ 2598.748913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_22d needs \>= 2 MDTs
[ 2598.983357] Lustre: DEBUG MARKER: SKIP: replay-dual test_22d needs >= 2 MDTs
[ 2599.287586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 20:37:13 \(1764880633\)
[ 2599.514022] Lustre: DEBUG MARKER: == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 20:37:13 (1764880633)
[ 2599.808623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_23a needs \>= 2 MDTs
[ 2600.035346] Lustre: DEBUG MARKER: SKIP: replay-dual test_23a needs >= 2 MDTs
[ 2600.346375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0\/M1, c2 mkdir d1 ========================================================== 20:37:14 \(1764880634\)
[ 2600.573744] Lustre: DEBUG MARKER: == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 20:37:14 (1764880634)
[ 2600.858713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_23b needs \>= 2 MDTs
[ 2601.104477] Lustre: DEBUG MARKER: SKIP: replay-dual test_23b needs >= 2 MDTs
[ 2601.417089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 20:37:15 \(1764880635\)
[ 2601.675739] Lustre: DEBUG MARKER: == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 20:37:15 (1764880635)
[ 2601.968709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_23c needs \>= 2 MDTs
[ 2602.192540] Lustre: DEBUG MARKER: SKIP: replay-dual test_23c needs >= 2 MDTs
[ 2602.499638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0\/M1, c2 mkdir d1 ========================================================== 20:37:16 \(1764880636\)
[ 2602.726989] Lustre: DEBUG MARKER: == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 20:37:16 (1764880636)
[ 2603.000889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_23d needs \>= 2 MDTs
[ 2603.224194] Lustre: DEBUG MARKER: SKIP: replay-dual test_23d needs >= 2 MDTs
[ 2603.527329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 24: reconstruct on non-existing object ========================================================== 20:37:17 \(1764880637\)
[ 2603.753966] Lustre: DEBUG MARKER: == replay-dual test 24: reconstruct on non-existing object ========================================================== 20:37:17 (1764880637)
[ 2690.343465] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 25: replay\|resend ==================== 20:38:44 \(1764880724\)
[ 2690.568335] Lustre: DEBUG MARKER: == replay-dual test 25: replay|resend ==================== 20:38:44 (1764880724)
[ 2709.895653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2710.121438] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2712.406794] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 2713.945259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 2714.058503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 2714.105443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 2714.269162] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2714.388593] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2714.390722] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2714.448536] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 2714.586020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
[ 2714.786464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL
IDLE state after 0 sec
[ 2714.828930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL
IDLE state after 0 sec
[ 2714.911763] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2715.137266] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL
[ 2715.139050] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL
[ 2718.215245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 26: dbench and tar with mds failover ========================================================== 20:39:12 \(1764880752\)
[ 2718.439213] Lustre: DEBUG MARKER: == replay-dual test 26: dbench and tar with mds failover ========================================================== 20:39:12 (1764880752)
[ 2718.633711] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
[ 2719.048341] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:
[ 2719.424100] Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
[ 2719.845298] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 2721.304306] Lustre: 13885:0:(mgc_request.c:1909:mgc_process_log()) MGC10.240.38.81@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 2721.306725] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[ 2721.306802] Lustre: Skipped 11 previous similar messages
[ 2721.384620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2721.388293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2721.622728] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2721.633597] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2724.309530] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 2724.832060] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 2730.088976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 1 times
[ 2730.420880] Lustre: DEBUG MARKER: test_26 fail mds1 1 times
[ 2804.539573] Autotest: Test running for 45 minutes (lustre-master_full-part-1_4681.175)
[ 2808.335128] LustreError: 49656:0:(ldlm_resource.c:978:ldlm_resource_complain()) MGC10.240.38.81@tcp: namespace resource [0x65727473756c:0x2:0x0].0x0 (ffff91fe95e8a0c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 2808.335171] Lustre: 13885:0:(mgc_request.c:1909:mgc_process_log()) MGC10.240.38.81@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 2832.957626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2833.205890] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2866.051330] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 2867.710245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2867.720442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2867.964475] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2867.998537] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2868.023127] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 2868.131063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2868.395074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2868.403073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2868.495113] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2868.651876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2868.663705] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2868.885541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2869.185052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2873.291976] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 2873.959992] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 2879.281227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 2 times
[ 2879.571228] Lustre: DEBUG MARKER: test_26 fail mds1 2 times
[ 2885.131048] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2885.131151] Lustre: Skipped 5 previous similar messages
[ 2900.489538] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 2900.489716] LustreError: Skipped 4 previous similar messages
[ 2900.493763] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a27e04b to 0xbe06701d5a30d39e
[ 2900.493877] Lustre: Skipped 3 previous similar messages
[ 2902.638659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2902.858997] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2912.186676] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 2913.863673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2913.866414] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2914.103560] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2914.103665] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2914.126995] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 2914.424801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2914.436402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2914.677874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2914.705600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2914.769440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2914.930233] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2915.342241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2915.636194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2919.642493] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 2920.193598] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 2925.422931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 3 times
[ 2925.786135] Lustre: DEBUG MARKER: test_26 fail mds1 3 times
[ 2948.579901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2948.807116] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2958.422774] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 2960.114710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2960.129518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2960.362374] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2960.362876] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2960.386454] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 2960.641438] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 2960.742403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2960.756336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2960.999396] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2961.058093] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2961.073987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2961.385974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 2961.681775] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2965.657586] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 2966.286061] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 2971.616410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 4 times
[ 2972.061100] Lustre: DEBUG MARKER: test_26 fail mds1 4 times
[ 2996.225574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 2996.435013] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3004.828267] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3006.418835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3006.424143] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3006.658782] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3006.658792] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3006.683376] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3007.006551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3007.046380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3007.063356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3007.380439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3007.388539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3007.391986] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3007.785692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3008.083226] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3012.061334] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 3012.623045] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 3017.825136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 5 times
[ 3018.141739] Lustre: DEBUG MARKER: test_26 fail mds1 5 times
[ 3041.790983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3042.007075] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3049.635049] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3051.312523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3051.312986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3051.567858] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3051.576291] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3051.597954] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3051.898052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3051.902712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3052.151322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3052.165276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3052.245952] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3052.408665] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3052.814616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3053.151081] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3057.159269] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 3057.748487] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 3062.950452] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 6 times
[ 3063.291014] Lustre: DEBUG MARKER: test_26 fail mds1 6 times
[ 3080.704489] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764881098/real 1764881098]  req@ffff91fe97d3d6c0 x1850608965906560/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764881114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3080.704708] Lustre: 6140:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3087.390082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3087.608461] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3095.797640] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3097.516995] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3097.517045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3097.755735] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3097.759022] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3097.779343] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3098.077481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3098.082031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3098.343530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3098.351537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3098.423849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3098.597185] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3099.051885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3099.296286] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3103.298744] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 3103.814465] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 3109.052433] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 7 times
[ 3109.404907] Lustre: DEBUG MARKER: test_26 fail mds1 7 times
[ 3133.390417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3133.628368] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3140.767462] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3142.428240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3142.428240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3142.665774] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3142.667216] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3142.692687] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3142.874735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3143.047140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3143.058445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3143.227397] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3143.346884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3143.365409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3143.634996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3143.969621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3147.993570] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 3148.671822] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 3153.890402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 8 times
[ 3154.263864] Lustre: DEBUG MARKER: test_26 fail mds1 8 times
[ 3163.123440] Autotest: Test running for 50 minutes (lustre-master_full-part-1_4681.175)
[ 3177.908121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3178.158657] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3184.263071] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3186.045671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3186.058955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3186.287172] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3186.296763] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3186.318883] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3186.644284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3186.694734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3186.714691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3187.014874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3187.033425] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3187.066171] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3187.482626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3187.791090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3191.772117] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 3192.357222] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 3197.680121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 9 times
[ 3198.042458] Lustre: DEBUG MARKER: test_26 fail mds1 9 times
[ 3222.032270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3222.245417] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3226.850772] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3228.674720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3228.685711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3228.945702] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3228.952908] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3228.972595] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3229.234057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3229.324030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3229.354657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3229.623641] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3229.639112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3229.697426] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3230.001059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3230.301940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3234.388507] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 3234.928027] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 3240.056157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 10 times
[ 3240.392520] Lustre: DEBUG MARKER: test_26 fail mds1 10 times
[ 3264.636526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3264.863199] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3270.726177] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3272.388165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3272.390355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3272.626856] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3272.628847] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3272.650967] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3272.859947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3273.000474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3273.012182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3273.217232] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3273.297737] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3273.338460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3273.648683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3273.935546] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3277.900826] Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
[ 3278.431855] Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
[ 3283.879065] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_26 fail mds1 11 times
[ 3284.643761] Lustre: DEBUG MARKER: test_26 fail mds1 11 times
[ 3314.093329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3314.331200] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3329.060454] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[ 3329.060619] Lustre: Skipped 32 previous similar messages
[ 3329.360826] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3331.065561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3331.065831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3331.337928] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3331.337927] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3331.364603] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3331.663353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3331.668242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3331.881457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3331.968259] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3332.007081] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3332.145023] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3332.732390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3333.049618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3338.219496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 20:49:32 \(1764881372\)
[ 3338.457906] Lustre: DEBUG MARKER: == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 20:49:32 (1764881372)
[ 3369.911893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3370.143144] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3436.841950] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3438.427284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 3438.459026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 3438.460204] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 3438.724261] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3438.780396] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3438.792050] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3438.812399] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3439.164083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
[ 3439.172136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL
IDLE state after 0 sec
[ 3439.224835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL
IDLE state after 0 sec
[ 3439.454891] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3439.529214] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL
[ 3439.544445] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL
[ 3444.063292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 29: replay vs update with the same xid ========================================================== 20:51:17 \(1764881477\)
[ 3444.306884] Lustre: DEBUG MARKER: == replay-dual test 29: replay vs update with the same xid ========================================================== 20:51:17 (1764881477)
[ 3444.584568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_29 needs \>= 2 MDTs
[ 3444.807489] Lustre: DEBUG MARKER: SKIP: replay-dual test_29 needs >= 2 MDTs
[ 3445.118271] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 20:51:18 \(1764881478\)
[ 3445.343889] Lustre: DEBUG MARKER: == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 20:51:18 (1764881478)
[ 3468.978854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3469.203545] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3480.189623] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3481.733491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3481.789104] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3481.813342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 3482.007347] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3482.091353] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3482.158920] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3482.183378] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3482.320474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3482.596359] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3482.605056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3482.611252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 3482.842515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3482.842599] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3485.319520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 20:51:59 \(1764881519\)
[ 3485.548202] Lustre: DEBUG MARKER: == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 20:51:59 (1764881519)
[ 3488.751613] Lustre: lustre-OST0000-osc-ffff91fe98965000: Connection to lustre-OST0000 (at 10.240.41.156@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3488.751715] Lustre: Skipped 21 previous similar messages
[ 3503.173514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3503.393593] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug -1 all
[ 3505.579186] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3507.245214] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 3507.283138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 3507.299542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
[ 3507.579104] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3507.593647] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3507.637709] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3507.659113] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 3507.901179] Autotest: Test running for 55 minutes (lustre-master_full-part-1_4681.175)
[ 3507.981815] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in IDLE
FULL state after 0 sec
[ 3508.022616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL
IDLE state after 0 sec
[ 3508.047299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL
IDLE state after 0 sec
[ 3508.294687] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE
[ 3508.356562] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL
[ 3508.361387] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL
[ 3511.211514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 32: gap in update llog shouldn\'t break recovery ========================================================== 20:52:25 \(1764881545\)
[ 3511.436778] Lustre: DEBUG MARKER: == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 20:52:25 (1764881545)
[ 3511.716296] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_32 needs \>= 2 MDTs
[ 3511.938479] Lustre: DEBUG MARKER: SKIP: replay-dual test_32 needs >= 2 MDTs
[ 3512.246753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 20:52:26 \(1764881546\)
[ 3512.473575] Lustre: DEBUG MARKER: == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 20:52:26 (1764881546)
[ 3512.762762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: replay-dual test_33 needs \>= 2 MDTs
[ 3512.984484] Lustre: DEBUG MARKER: SKIP: replay-dual test_33 needs >= 2 MDTs
[ 3513.286865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test complete, duration 2922 sec ========== 20:52:27 \(1764881547\)
[ 3513.511745] Lustre: DEBUG MARKER: == replay-dual test complete, duration 2922 sec ========== 20:52:27 (1764881547)
[ 3513.812611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === replay-dual: start cleanup 20:52:27 \(1764881547\) ===
[ 3514.038710] Lustre: DEBUG MARKER: === replay-dual: start cleanup 20:52:27 (1764881547) ===
[ 3515.275257] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[ 3515.651438] LustreError: 68818:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91ff828fc000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3515.651532] LustreError: 68818:0:(lov_obd.c:780:lov_cleanup()) Skipped 6 previous similar messages
[ 3515.655596] LustreError: 68818:0:(obd_class.h:478:obd_check_dev()) Device 20 not setup
[ 3515.655701] LustreError: 68818:0:(obd_class.h:478:obd_check_dev()) Skipped 17 previous similar messages
[ 3515.675666] Lustre: Unmounted lustre-client
[ 3516.327416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === replay-dual: finish cleanup 20:52:30 \(1764881550\) ===
[ 3516.551102] Lustre: DEBUG MARKER: === replay-dual: finish cleanup 20:52:30 (1764881550) ===
[ 3516.867220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-flr ============----- Thu Dec  4 08:52:30 PM UTC 2025
[ 3517.097484] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-flr ============----- Thu Dec 4 08:52:30 PM UTC 2025
[ 3522.727464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 6 201 44c
[ 3522.950692] Lustre: DEBUG MARKER: excepting tests: 6 201 44c
[ 3523.242275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity-flr: start setup 20:52:37 \(1764881557\) ===
[ 3523.468641] Lustre: DEBUG MARKER: === sanity-flr: start setup 20:52:37 (1764881557) ===
[ 3523.703045] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3525.258426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 3525.318755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 3525.359341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 3525.547959] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 3525.642734] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 3525.664909] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 3529.273087] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 3529.698686] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 3537.160385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[ 3537.383432] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3537.657616] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[ 3540.329859] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 3541.902412] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3541.904206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3542.141335] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3542.141346] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3542.607326] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3542.617045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3542.871222] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3542.944774] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3544.290365] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity-flr: finish setup 20:52:58 \(1764881578\) ===
[ 3544.522127] Lustre: DEBUG MARKER: === sanity-flr: finish setup 20:52:58 (1764881578) ===
[ 3544.875064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0a: lfs mirror create with -N option == 20:52:58 \(1764881578\)
[ 3545.099681] Lustre: DEBUG MARKER: == sanity-flr test 0a: lfs mirror create with -N option == 20:52:58 (1764881578)
[ 3547.753956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:53:01 \(1764881581\)
[ 3547.986203] Lustre: DEBUG MARKER: == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:53:01 (1764881581)
[ 3582.003940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:53:35 \(1764881615\)
[ 3582.232051] Lustre: DEBUG MARKER: == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:53:35 (1764881615)
[ 3619.314650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0d: lfs mirror extend with -N option == 20:54:13 \(1764881653\)
[ 3619.539322] Lustre: DEBUG MARKER: == sanity-flr test 0d: lfs mirror extend with -N option == 20:54:13 (1764881653)
[ 3623.624633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:54:17 \(1764881657\)
[ 3623.849432] Lustre: DEBUG MARKER: == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:54:17 (1764881657)
[ 3642.455535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:54:36 \(1764881676\)
[ 3642.683901] Lustre: DEBUG MARKER: == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:54:36 (1764881676)
[ 3662.571727] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0g: lfs mirror create flags support === 20:54:56 \(1764881696\)
[ 3662.796895] Lustre: DEBUG MARKER: == sanity-flr test 0g: lfs mirror create flags support === 20:54:56 (1764881696)
[ 3664.259541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:54:58 \(1764881698\)
[ 3664.481741] Lustre: DEBUG MARKER: == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:54:58 (1764881698)
[ 3666.341962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0j: test lfs mirror read\/write commands ========================================================== 20:55:00 \(1764881700\)
[ 3666.565720] Lustre: DEBUG MARKER: == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:55:00 (1764881700)
[ 3668.215981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:55:02 \(1764881702\)
[ 3668.444668] Lustre: DEBUG MARKER: == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:55:02 (1764881702)
[ 3668.766576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity-flr test_0k: @@@@@@ FAIL: mirror extend failed 
[ 3668.985803] Lustre: DEBUG MARKER: sanity-flr test_0k: @@@@@@ FAIL: mirror extend failed
[ 3669.259946] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_4681_175_3e4a9bbf-3bb6-4eb4-addf-bb27972412b0//sanity-flr.test_0k.debug_log.$(hostname -s).1764881703.log;
		dmesg > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_46
[ 3670.679534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 1: create components with setstripe options ========================================================== 20:55:04 \(1764881704\)
[ 3670.906955] Lustre: DEBUG MARKER: == sanity-flr test 1: create components with setstripe options ========================================================== 20:55:04 (1764881704)
[ 3672.507108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 2: create components from existing files ========================================================== 20:55:06 \(1764881706\)
[ 3672.729928] Lustre: DEBUG MARKER: == sanity-flr test 2: create components from existing files ========================================================== 20:55:06 (1764881706)
[ 3674.083599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:55:07 \(1764881707\)
[ 3674.308909] Lustre: DEBUG MARKER: == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:55:07 (1764881707)
[ 3674.590809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_3 need \>= 2 MDTs
[ 3674.809001] Lustre: DEBUG MARKER: SKIP: sanity-flr test_3 need >= 2 MDTs
[ 3675.132883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:55:09 \(1764881709\)
[ 3675.354389] Lustre: DEBUG MARKER: == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:55:09 (1764881709)
[ 3676.710474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:55:10 \(1764881710\)
[ 3676.939656] Lustre: DEBUG MARKER: == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:55:10 (1764881710)
[ 3678.232536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a \(base 6\)
[ 3678.457076] Lustre: DEBUG MARKER: SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6)
[ 3678.741515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b \(base 6\)
[ 3678.962978] Lustre: DEBUG MARKER: SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6)
[ 3679.262485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 7: Create mirror with DoM component === 20:55:13 \(1764881713\)
[ 3679.487816] Lustre: DEBUG MARKER: == sanity-flr test 7: Create mirror with DoM component === 20:55:13 (1764881713)
[ 3680.778928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:55:14 \(1764881714\)
[ 3681.005207] Lustre: DEBUG MARKER: == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:55:14 (1764881714)
[ 3683.578579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:55:17 \(1764881717\)
[ 3683.803711] Lustre: DEBUG MARKER: == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:55:17 (1764881717)
[ 3685.889934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:55:19 \(1764881719\)
[ 3686.115654] Lustre: DEBUG MARKER: == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:55:19 (1764881719)
[ 3688.758268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3688.983332] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3691.283341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 2 sec
[ 3691.504109] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 2 sec
[ 3702.573641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3702.796743] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3705.884621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3706.100421] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3708.592078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3708.825349] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3712.134490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 3 sec
[ 3712.347205] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 3 sec
[ 3719.437124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3719.663788] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3722.928242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3723.147899] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3724.450617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:55:58 \(1764881758\)
[ 3724.679999] Lustre: DEBUG MARKER: == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:55:58 (1764881758)
[ 3727.675391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3727.908686] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3728.190224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3728.408654] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3734.325658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3734.542734] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3737.698578] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3737.914155] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3739.216374] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:56:13 \(1764881773\)
[ 3739.441134] Lustre: DEBUG MARKER: == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:56:13 (1764881773)
[ 3742.171538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3742.396612] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3742.673734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3742.892263] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3748.856342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3749.081324] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3752.142405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3752.360940] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3754.845567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3755.070854] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3758.379498] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 3 sec
[ 3758.600399] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 3 sec
[ 3764.746693] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3764.966819] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3768.091427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3768.310628] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3769.634536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:56:43 \(1764881803\)
[ 3769.860167] Lustre: DEBUG MARKER: == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:56:43 (1764881803)
[ 3776.126892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3776.365519] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3778.668493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 2 sec
[ 3778.886702] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 2 sec
[ 3785.815715] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3786.040867] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3789.148485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3789.361787] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3791.838521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3792.067789] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3794.442391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 2 sec
[ 3794.670602] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 2 sec
[ 3801.362228] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3801.582477] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3804.808917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3805.038679] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3806.045069] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(FULL\|IDLE\) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3806.272441] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3806.555028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in FULL state after 0 sec
[ 3806.780132] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in FULL state after 0 sec
[ 3808.136409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:57:22 \(1764881842\)
[ 3808.364453] Lustre: DEBUG MARKER: == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:57:22 (1764881842)
[ 3816.375719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3816.601762] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3819.918841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 3 sec
[ 3820.147567] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 3 sec
[ 3891.436209] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3891.652322] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3894.761500] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3894.990757] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3895.978348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(FULL\|IDLE\) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3896.206597] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3896.486701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in FULL state after 0 sec
[ 3896.710386] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in FULL state after 0 sec
[ 3898.030897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:58:51 \(1764881931\)
[ 3898.253348] Lustre: DEBUG MARKER: == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:58:51 (1764881931)
[ 3902.845479] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3903.070348] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3903.344915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3903.579100] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3904.524655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0002-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3904.750804] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0002-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3907.048915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 2 sec
[ 3907.270048] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 2 sec
[ 3913.275442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3913.490711] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3920.408997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3920.632531] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3923.890879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3924.116290] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3926.450377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0002.recovery_status 1475
[ 3926.667957] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0002.recovery_status 1475
[ 3928.003992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:59:21 \(1764881961\)
[ 3928.233648] Lustre: DEBUG MARKER: == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:59:21 (1764881961)
[ 3932.606567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3932.830294] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3933.108570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3933.331196] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3934.278093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state \(DISCONN\|IDLE\) osc.lustre-OST0002-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3934.500054] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0002-osc-ffff91411e6d8800.ost_server_uuid 50
[ 3934.776171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3934.995085] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-ffff91411e6d8800.ost_server_uuid in DISCONN state after 0 sec
[ 3940.940945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3941.157786] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3948.086697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3948.307337] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3951.409313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3951.636449] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3954.053615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0002.recovery_status 1475
[ 3954.269182] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0002.recovery_status 1475
[ 3955.605615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 35: allow to write to mirrored files == 20:59:49 \(1764881989\)
[ 3955.839553] Lustre: DEBUG MARKER: == sanity-flr test 35: allow to write to mirrored files == 20:59:49 (1764881989)
[ 3957.342951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 36a: write to mirrored files ========== 20:59:51 \(1764881991\)
[ 3957.563344] Lustre: DEBUG MARKER: == sanity-flr test 36a: write to mirrored files ========== 20:59:51 (1764881991)
[ 3970.458477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:00:04 \(1764882004\)
[ 3970.684963] Lustre: DEBUG MARKER: == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:00:04 (1764882004)
[ 3970.962938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_36b Need OST version at least 2.15.50
[ 3971.182160] Lustre: DEBUG MARKER: SKIP: sanity-flr test_36b Need OST version at least 2.15.50
[ 3971.486144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 36c: change prefer mirror during write shouldn\'t hung ========================================================== 21:00:05 \(1764882005\)
[ 3971.711173] Lustre: DEBUG MARKER: == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:00:05 (1764882005)
[ 3971.991640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_36c Need OST version at least 2.15.50
[ 3972.211271] Lustre: DEBUG MARKER: SKIP: sanity-flr test_36c Need OST version at least 2.15.50
[ 3972.514220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 36d: write\/punch FLR file update OST layout version ========================================================== 21:00:06 \(1764882006\)
[ 3972.742300] Lustre: DEBUG MARKER: == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:00:06 (1764882006)
[ 3973.023098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_36d Need OST version at least 2.15.50
[ 3973.246302] Lustre: DEBUG MARKER: SKIP: sanity-flr test_36d Need OST version at least 2.15.50
[ 3973.551733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 37: mirror I\/O API verification ======= 21:00:07 \(1764882007\)
[ 3973.774675] Lustre: DEBUG MARKER: == sanity-flr test 37: mirror I/O API verification ======= 21:00:07 (1764882007)
[ 3980.337248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 38: resync ============================ 21:00:14 \(1764882014\)
[ 3980.558785] Lustre: DEBUG MARKER: == sanity-flr test 38: resync ============================ 21:00:14 (1764882014)
[ 3983.426442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 39: check FLR+PFL \(a.k.a. PFLR\) creation ========================================================== 21:00:17 \(1764882017\)
[ 3983.653562] Lustre: DEBUG MARKER: == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:00:17 (1764882017)
[ 3984.943556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:00:18 \(1764882018\)
[ 3985.169339] Lustre: DEBUG MARKER: == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:00:18 (1764882018)
[ 3986.788697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 41: lfs mirror resync check =========== 21:00:20 \(1764882020\)
[ 3987.011455] Lustre: DEBUG MARKER: == sanity-flr test 41: lfs mirror resync check =========== 21:00:20 (1764882020)
[ 3990.600747] Autotest: Test running for 65 minutes (lustre-master_full-part-1_4681.175)
[ 3991.221923] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 42: lfs mirror verify ================= 21:00:25 \(1764882025\)
[ 3991.445221] Lustre: DEBUG MARKER: == sanity-flr test 42: lfs mirror verify ================= 21:00:25 (1764882025)
[ 3998.877920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 43a: mirror pick on write ============= 21:00:32 \(1764882032\)
[ 3999.098321] Lustre: DEBUG MARKER: == sanity-flr test 43a: mirror pick on write ============= 21:00:32 (1764882032)
[ 4075.922656] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:01:49 \(1764882109\)
[ 4076.158462] Lustre: DEBUG MARKER: == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:01:49 (1764882109)
[ 4077.804493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 44a: lfs mirror split check =========== 21:01:51 \(1764882111\)
[ 4078.025796] Lustre: DEBUG MARKER: == sanity-flr test 44a: lfs mirror split check =========== 21:01:51 (1764882111)
[ 4078.296336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_44a needs \>= 2 MDTs
[ 4078.515787] Lustre: DEBUG MARKER: SKIP: sanity-flr test_44a needs >= 2 MDTs
[ 4078.827602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:01:52 \(1764882112\)
[ 4079.049611] Lustre: DEBUG MARKER: == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:01:52 (1764882112)
[ 4080.682282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c
[ 4080.904017] Lustre: DEBUG MARKER: SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c
[ 4081.208753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:01:55 \(1764882115\)
[ 4081.430824] Lustre: DEBUG MARKER: == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:01:55 (1764882115)
[ 4084.292990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:01:58 \(1764882118\)
[ 4084.516522] Lustre: DEBUG MARKER: == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:01:58 (1764882118)
[ 4084.807627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_44e Need MDS version \>= 2.15.50.155 for SOM tunable
[ 4085.031086] Lustre: DEBUG MARKER: SKIP: sanity-flr test_44e Need MDS version >= 2.15.50.155 for SOM tunable
[ 4085.363330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 45: Verify setstripe\/getstripe with YAML with FLR file ========================================================== 21:01:59 \(1764882119\)
[ 4085.586003] Lustre: DEBUG MARKER: == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:01:59 (1764882119)
[ 4092.055645] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 46: Verify setstripe --copy option ==== 21:02:05 \(1764882125\)
[ 4092.275404] Lustre: DEBUG MARKER: == sanity-flr test 46: Verify setstripe --copy option ==== 21:02:05 (1764882125)
[ 4098.316746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 47: Verify mirror obj alloc =========== 21:02:12 \(1764882132\)
[ 4098.539773] Lustre: DEBUG MARKER: == sanity-flr test 47: Verify mirror obj alloc =========== 21:02:12 (1764882132)
[ 4100.740065] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 48: Verify snapshot mirror ============ 21:02:14 \(1764882134\)
[ 4100.961829] Lustre: DEBUG MARKER: == sanity-flr test 48: Verify snapshot mirror ============ 21:02:14 (1764882134)
[ 4103.300332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:02:17 \(1764882137\)
[ 4103.521239] Lustre: DEBUG MARKER: == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:02:17 (1764882137)
[ 4105.065024] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:02:18 \(1764882138\)
[ 4105.281545] Lustre: DEBUG MARKER: == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:02:18 (1764882138)
[ 4108.241280] Lustre: lustre-MDT0000-mdc-ffff91fe98965000: Connection to lustre-MDT0000 (at 10.240.38.81@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4129.743770] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764882147/real 1764882147]  req@ffff91fe97d3c000 x1850608966124416/t0(0) o400->MGC10.240.38.81@tcp@10.240.38.81@tcp:26/25 lens 224/224 e 0 to 1 dl 1764882163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4129.743898] Lustre: 6141:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 4129.744025] LustreError: MGC10.240.38.81@tcp: Connection to MGS (at 10.240.38.81@tcp) was lost; in progress operations using this service will fail
[ 4129.744092] LustreError: Skipped 10 previous similar messages
[ 4133.608347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4133.825322] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4138.959850] Lustre: Evicted from MGS (at 10.240.38.81@tcp) after server handle changed from 0xbe06701d5a51337b to 0xbe06701d5a527440
[ 4138.959980] Lustre: Skipped 10 previous similar messages
[ 4138.963197] Lustre: MGC10.240.38.81@tcp: Connection restored to 10.240.38.81@tcp (at 10.240.38.81@tcp)
[ 4138.963335] Lustre: Skipped 4 previous similar messages
[ 4139.696715] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 4141.247309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 4141.283038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 4141.314909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
[ 4141.559031] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4141.591590] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4141.632639] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4141.653075] Lustre: DEBUG MARKER: lctl get_param -n at_max
[ 4141.935306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 4141.988068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 4142.025841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
[ 4142.194404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4142.307184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4142.307192] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4144.845436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 50a: mirror extend\/copy preserves sparseness ========================================================== 21:02:58 \(1764882178\)
[ 4145.070713] Lustre: DEBUG MARKER: == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:02:58 (1764882178)
[ 4145.352039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_50a Need OST version at least 2.15.58 for unaligned DIO
[ 4145.575367] Lustre: DEBUG MARKER: SKIP: sanity-flr test_50a Need OST version at least 2.15.58 for unaligned DIO
[ 4145.883817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 50b: mirror rsync handles sparseness == 21:02:59 \(1764882179\)
[ 4146.105639] Lustre: DEBUG MARKER: == sanity-flr test 50b: mirror rsync handles sparseness == 21:02:59 (1764882179)
[ 4148.024546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 50c: punch_hole\/mmap_write stale other mirrors ========================================================== 21:03:01 \(1764882181\)
[ 4148.247001] Lustre: DEBUG MARKER: == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:03:01 (1764882181)
[ 4149.834885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 50d: mirror rsync keep holes ========== 21:03:03 \(1764882183\)
[ 4150.056947] Lustre: DEBUG MARKER: == sanity-flr test 50d: mirror rsync keep holes ========== 21:03:03 (1764882183)
[ 4151.616855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:03:05 \(1764882185\)
[ 4151.842317] Lustre: DEBUG MARKER: == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:03:05 (1764882185)
[ 4153.533020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 60b: mirror merge\/split cancel client\'s in-memory layout gen ========================================================== 21:03:07 \(1764882187\)
[ 4153.779716] Lustre: DEBUG MARKER: == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:03:07 (1764882187)
[ 4156.652623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:03:10 \(1764882190\)
[ 4156.875466] Lustre: DEBUG MARKER: == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:03:10 (1764882190)
[ 4174.631583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:03:28 \(1764882208\)
[ 4174.852578] Lustre: DEBUG MARKER: == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:03:28 (1764882208)
[ 4191.415762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:03:45 \(1764882225\)
[ 4191.645661] Lustre: DEBUG MARKER: == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:03:45 (1764882225)
[ 4208.311874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 62: read\/write with unknown type of mirror ========================================================== 21:04:02 \(1764882242\)
[ 4208.537337] Lustre: DEBUG MARKER: == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:04:02 (1764882242)
[ 4209.995292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 70: mirror create and split race ====== 21:04:03 \(1764882243\)
[ 4210.211688] Lustre: DEBUG MARKER: == sanity-flr test 70: mirror create and split race ====== 21:04:03 (1764882243)
[ 4271.520304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 70a: flr mode fsx test ================ 21:05:05 \(1764882305\)
[ 4271.747528] Lustre: DEBUG MARKER: == sanity-flr test 70a: flr mode fsx test ================ 21:05:05 (1764882305)
[ 4296.791935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file \/mnt\/lustre\/d70a.sanity-flr\/f70a.sanity-flr failed 
[ 4297.020487] Lustre: DEBUG MARKER: sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed
[ 4297.314768] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_4681_175_3e4a9bbf-3bb6-4eb4-addf-bb27972412b0//sanity-flr.test_70a.debug_log.$(hostname -s).1764882331.log;
		dmesg > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_4
[ 4297.924546] Autotest: Test running for 70 minutes (lustre-master_full-part-1_4681.175)
[ 4384.277501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 71: check mirror extend parent fid ==== 21:06:58 \(1764882418\)
[ 4384.499684] Lustre: DEBUG MARKER: == sanity-flr test 71: check mirror extend parent fid ==== 21:06:58 (1764882418)
[ 4395.141708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4395.356985] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4397.787459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  \*\* lfs fid2path \/mnt\/lustre ::
[ 4398.012755] Lustre: DEBUG MARKER: ** lfs fid2path /mnt/lustre ::
[ 4398.306259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity-flr test_71: @@@@@@ FAIL: cannot find parent :: of OST object [0x100000000:0x2454:0x0] 
[ 4398.535464] Lustre: DEBUG MARKER: sanity-flr test_71: @@@@@@ FAIL: cannot find parent :: of OST object [0x100000000:0x2454:0x0]
[ 4398.822456] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_4681_175_3e4a9bbf-3bb6-4eb4-addf-bb27972412b0//sanity-flr.test_71.debug_log.$(hostname -s).1764882432.log;
		dmesg > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_46
[ 4399.861546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 71: check append on FLR file ========== 21:07:13 \(1764882433\)
[ 4400.088267] Lustre: DEBUG MARKER: == sanity-flr test 71: check append on FLR file ========== 21:07:13 (1764882433)
[ 4401.894157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 200a: stress test ===================== 21:07:15 \(1764882435\)
[ 4402.115858] Lustre: DEBUG MARKER: == sanity-flr test 200a: stress test ===================== 21:07:15 (1764882435)
[ 4559.692688] Autotest: Test running for 75 minutes (lustre-master_full-part-1_4681.175)
[ 4767.170336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:13:21 \(1764882801\)
[ 4767.392225] Lustre: DEBUG MARKER: == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:13:21 (1764882801)
[ 4860.592866] Autotest: Test running for 80 minutes (lustre-master_full-part-1_4681.175)
[ 5161.598298] Autotest: Test running for 85 minutes (lustre-master_full-part-1_4681.175)
[ 5171.510344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:20:05 \(1764883205\)
[ 5171.736307] Lustre: DEBUG MARKER: == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:20:05 (1764883205)
[ 5172.013249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_200c Need MDS version at least 2.15.53
[ 5172.238732] Lustre: DEBUG MARKER: SKIP: sanity-flr test_200c Need MDS version at least 2.15.53
[ 5172.528157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201
[ 5172.752653] Lustre: DEBUG MARKER: SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201
[ 5173.057458] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:20:07 \(1764883207\)
[ 5173.282813] Lustre: DEBUG MARKER: == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:20:07 (1764883207)
[ 5174.659134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 203: mirror file preserve mirror ID === 21:20:08 \(1764883208\)
[ 5174.880014] Lustre: DEBUG MARKER: == sanity-flr test 203: mirror file preserve mirror ID === 21:20:08 (1764883208)
[ 5176.268422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 204a: FLR write\/stale\/resync tests with self-extending mirror ========================================================== 21:20:10 \(1764883210\)
[ 5176.493612] Lustre: DEBUG MARKER: == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:20:10 (1764883210)
[ 5178.260379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 204b: FLR write\/stale\/resync tests with self-extending primary ========================================================== 21:20:12 \(1764883212\)
[ 5178.482357] Lustre: DEBUG MARKER: == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:20:12 (1764883212)
[ 5180.201550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 204c: FLR write\/stale\/resync test with component removal ========================================================== 21:20:14 \(1764883214\)
[ 5180.426121] Lustre: DEBUG MARKER: == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:20:14 (1764883214)
[ 5203.425402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 204d: FLR write\/stale\/resync sel test with repeated comp ========================================================== 21:20:37 \(1764883237\)
[ 5203.653957] Lustre: DEBUG MARKER: == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:20:37 (1764883237)
[ 5237.596064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 204e: FLR write\/stale\/resync sel test with repeated comp ========================================================== 21:21:11 \(1764883271\)
[ 5237.817301] Lustre: DEBUG MARKER: == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:21:11 (1764883271)
[ 5271.731064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 204f: FLR write\/stale\/resync sel w\/forced extension ========================================================== 21:21:45 \(1764883305\)
[ 5271.956689] Lustre: DEBUG MARKER: == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:21:45 (1764883305)
[ 5317.825788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:22:31 \(1764883351\)
[ 5318.054475] Lustre: DEBUG MARKER: == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:22:31 (1764883351)
[ 5319.480299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:22:33 \(1764883353\)
[ 5319.701925] Lustre: DEBUG MARKER: == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:22:33 (1764883353)
[ 5319.986048] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-flr test_205b Need MDS \> v2_15_61-245-g37e1316050 to test nocompr flag
[ 5320.208708] Lustre: DEBUG MARKER: SKIP: sanity-flr test_205b Need MDS > v2_15_61-245-g37e1316050 to test nocompr flag
[ 5320.535858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:22:34 \(1764883354\)
[ 5320.761313] Lustre: DEBUG MARKER: == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:22:34 (1764883354)
[ 5334.766710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:22:48 \(1764883368\)
[ 5334.989255] Lustre: DEBUG MARKER: == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:22:48 (1764883368)
[ 5336.777081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:22:50 \(1764883370\)
[ 5337.003693] Lustre: DEBUG MARKER: == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:22:50 (1764883370)
[ 5341.180416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set OST0000 non-rotational
[ 5341.397015] Lustre: DEBUG MARKER: set OST0000 non-rotational
[ 5355.104324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set OST0002 and OST0003 non-rotational, two fast OSTs is better
[ 5355.329898] Lustre: DEBUG MARKER: set OST0002 and OST0003 non-rotational, two fast OSTs is better
[ 5368.825801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set mirror 1 on OST0001 preferred
[ 5369.064124] Lustre: DEBUG MARKER: set mirror 1 on OST0001 preferred
[ 5385.380948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:23:39 \(1764883419\)
[ 5385.613280] Lustre: DEBUG MARKER: == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:23:39 (1764883419)
[ 5389.804609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set OST0000 non-rotational
[ 5390.028581] Lustre: DEBUG MARKER: set OST0000 non-rotational
[ 5404.165768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set OST0002 and OST0003 non-rotational, two fast OSTs is better
[ 5404.389194] Lustre: DEBUG MARKER: set OST0002 and OST0003 non-rotational, two fast OSTs is better
[ 5418.623840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set mirror 1 on OST0001 preferred
[ 5418.848972] Lustre: DEBUG MARKER: set mirror 1 on OST0001 preferred
[ 5435.234136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:24:29 \(1764883469\)
[ 5435.465803] Lustre: DEBUG MARKER: == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:24:29 (1764883469)
[ 5439.214500] Lustre: DEBUG MARKER: /usr/sbin/lctl mark replicated file created
[ 5439.434596] Lustre: DEBUG MARKER: replicated file created
[ 5439.727009] Lustre: DEBUG MARKER: /usr/sbin/lctl mark open\(O_RDONLY\) and first read from OST
[ 5439.946525] Lustre: DEBUG MARKER: open(O_RDONLY) and first read from OST
[ 5441.226601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark first read complete
[ 5441.451594] Lustre: DEBUG MARKER: first read complete
[ 5441.748207] Lustre: DEBUG MARKER: /usr/sbin/lctl mark BBBB written which made replica on ost1 stale
[ 5441.971372] Lustre: DEBUG MARKER: BBBB written which made replica on ost1 stale
[ 5442.249184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark fast read from pagecache in the original process
[ 5442.479617] Lustre: DEBUG MARKER: fast read from pagecache in the original process
[ 5443.766926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark read via \/mnt\/lustre2 new open\(2\)
[ 5443.983311] Lustre: DEBUG MARKER: read via /mnt/lustre2 new open(2)
[ 5444.276118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark fast read from pagecache after 5s in the original process
[ 5444.496190] Lustre: DEBUG MARKER: fast read from pagecache after 5s in the original process
[ 5449.783062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark read via new open\(2\)
[ 5450.003377] Lustre: DEBUG MARKER: read via new open(2)
[ 5453.878844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:24:47 \(1764883487\)
[ 5454.102409] Lustre: DEBUG MARKER: == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:24:47 (1764883487)
[ 5455.456852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 210a: handle broken mirrored lovea ==== 21:24:49 \(1764883489\)
[ 5455.677846] Lustre: DEBUG MARKER: == sanity-flr test 210a: handle broken mirrored lovea ==== 21:24:49 (1764883489)
[ 5457.484906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 210b: handle broken mirrored lovea \(unlink\) ========================================================== 21:24:51 \(1764883491\)
[ 5457.706331] Lustre: DEBUG MARKER: == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:24:51 (1764883491)
[ 5469.845011] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:25:03 \(1764883503\)
[ 5470.066953] Lustre: DEBUG MARKER: == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:25:03 (1764883503)
[ 5471.879212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:25:05 \(1764883505\)
[ 5472.109664] Lustre: DEBUG MARKER: == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:25:05 (1764883505)
[ 5473.647057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:25:07 \(1764883507\)
[ 5473.871303] Lustre: DEBUG MARKER: == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:25:07 (1764883507)
[ 5474.379003] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity-flr test_212b: @@@@@@ FAIL: verify should fail when mirrors are inconsistent 
[ 5474.602198] Lustre: DEBUG MARKER: sanity-flr test_212b: @@@@@@ FAIL: verify should fail when mirrors are inconsistent
[ 5474.877282] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_4681_175_3e4a9bbf-3bb6-4eb4-addf-bb27972412b0//sanity-flr.test_212b.debug_log.$(hostname -s).1764883508.log;
		dmesg > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_
[ 5476.019965] Autotest: Test running for 90 minutes (lustre-master_full-part-1_4681.175)
[ 5476.726040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-flr test complete, duration 1959 sec =========== 21:25:10 \(1764883510\)
[ 5476.944817] Lustre: DEBUG MARKER: == sanity-flr test complete, duration 1959 sec =========== 21:25:10 (1764883510)
[ 5477.252407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity-flr: start cleanup 21:25:11 \(1764883511\) ===
[ 5477.473855] Lustre: DEBUG MARKER: === sanity-flr: start cleanup 21:25:11 (1764883511) ===
[ 5479.175383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity-flr: finish cleanup 21:25:13 \(1764883513\) ===
[ 5479.403699] Lustre: DEBUG MARKER: === sanity-flr: finish cleanup 21:25:13 (1764883513) ===
[ 5479.712627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-lsnapshot ============----- Thu Dec  4 09:25:13 PM UTC 2025
[ 5479.928724] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lsnapshot ============----- Thu Dec 4 09:25:13 PM UTC 2025
[ 5485.557512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 
[ 5485.788475] Lustre: DEBUG MARKER: excepting tests:
[ 5486.088759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lsnapshot  ZFS only test
[ 5486.308566] Lustre: DEBUG MARKER: SKIP: sanity-lsnapshot ZFS only test
[ 5486.613951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: mmp ============----- Thu Dec  4 09:25:20 PM UTC 2025
[ 5486.838888] Lustre: DEBUG MARKER: -----============= acceptance-small: mmp ============----- Thu Dec 4 09:25:20 PM UTC 2025
[ 5492.497725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 
[ 5492.721169] Lustre: DEBUG MARKER: excepting tests:
[ 5493.040959] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre opts:;
lsof /mnt/lustre || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre | sort -u);
	if [ -n "
[ 5493.384075] LustreError: 94656:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91fe98965000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5493.384163] LustreError: 94656:0:(lov_obd.c:780:lov_cleanup()) Skipped 6 previous similar messages
[ 5493.388731] LustreError: 94656:0:(obd_class.h:478:obd_check_dev()) Device 10 not setup
[ 5493.388862] LustreError: 94656:0:(obd_class.h:478:obd_check_dev()) Skipped 15 previous similar messages
[ 5493.408191] Lustre: Unmounted lustre-client
[ 5493.597273] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[ 5530.156652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MMP is not enabled on MDS, enabling it manually...
[ 5530.383683] Lustre: DEBUG MARKER: MMP is not enabled on MDS, enabling it manually...
[ 5531.365908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MMP is not enabled on OSS, enabling it manually...
[ 5531.595695] Lustre: DEBUG MARKER: MMP is not enabled on OSS, enabling it manually...
[ 5532.984654] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 1: two mounts at the same time =============== 21:26:06 \(1764883566\)
[ 5533.213120] Lustre: DEBUG MARKER: == mmp test 1: two mounts at the same time =============== 21:26:06 (1764883566)
[ 5533.480435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: mmp test_1 failover pair is needed
[ 5533.707866] Lustre: DEBUG MARKER: SKIP: mmp test_1 failover pair is needed
[ 5534.007034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 2: one mount delayed by mmp update interval == 21:26:07 \(1764883567\)
[ 5534.226432] Lustre: DEBUG MARKER: == mmp test 2: one mount delayed by mmp update interval == 21:26:07 (1764883567)
[ 5534.492943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: mmp test_2 failover pair is needed
[ 5534.725959] Lustre: DEBUG MARKER: SKIP: mmp test_2 failover pair is needed
[ 5535.022726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 3: one mount delayed by 2x mmp check interval ========================================================== 21:26:08 \(1764883568\)
[ 5535.242789] Lustre: DEBUG MARKER: == mmp test 3: one mount delayed by 2x mmp check interval ========================================================== 21:26:08 (1764883568)
[ 5535.510190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: mmp test_3 failover pair is needed
[ 5535.729609] Lustre: DEBUG MARKER: SKIP: mmp test_3 failover pair is needed
[ 5536.027043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 4: one mount delayed by \> 2x mmp check interval ========================================================== 21:26:10 \(1764883570\)
[ 5536.248929] Lustre: DEBUG MARKER: == mmp test 4: one mount delayed by > 2x mmp check interval ========================================================== 21:26:10 (1764883570)
[ 5536.516160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: mmp test_4 failover pair is needed
[ 5536.736876] Lustre: DEBUG MARKER: SKIP: mmp test_4 failover pair is needed
[ 5537.034642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 5: mount during unmount of the first filesystem ========================================================== 21:26:11 \(1764883571\)
[ 5537.253915] Lustre: DEBUG MARKER: == mmp test 5: mount during unmount of the first filesystem ========================================================== 21:26:11 (1764883571)
[ 5537.524614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: mmp test_5 failover pair is needed
[ 5537.745015] Lustre: DEBUG MARKER: SKIP: mmp test_5 failover pair is needed
[ 5538.042715] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 6: mount after clean unmount ================= 21:26:12 \(1764883572\)
[ 5538.268852] Lustre: DEBUG MARKER: == mmp test 6: mount after clean unmount ================= 21:26:12 (1764883572)
[ 5538.538601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: mmp test_6 failover pair is needed
[ 5538.758215] Lustre: DEBUG MARKER: SKIP: mmp test_6 failover pair is needed
[ 5539.058634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 7: mount after reboot ======================== 21:26:13 \(1764883573\)
[ 5539.280112] Lustre: DEBUG MARKER: == mmp test 7: mount after reboot ======================== 21:26:13 (1764883573)
[ 5539.545435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: mmp test_7 failover pair is needed
[ 5539.766909] Lustre: DEBUG MARKER: SKIP: mmp test_7 failover pair is needed
[ 5540.081060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 8: mount during e2fsck ======================= 21:26:14 \(1764883574\)
[ 5540.305311] Lustre: DEBUG MARKER: == mmp test 8: mount during e2fsck ======================= 21:26:14 (1764883574)
[ 5779.937427] Autotest: Test running for 95 minutes (lustre-master_full-part-1_4681.175)
[ 5811.543979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 9: mount after aborted e2fsck ================ 21:30:45 \(1764883845\)
[ 5811.766727] Lustre: DEBUG MARKER: == mmp test 9: mount after aborted e2fsck ================ 21:30:45 (1764883845)
[ 5818.225212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5818.443990] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5826.750383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5826.963811] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5854.992439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/vg_Role_MDS\/mdt1 on mds1...
[ 5855.215975] Lustre: DEBUG MARKER: Mounting /dev/vg_Role_MDS/mdt1 on mds1...
[ 5870.410552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/vg_Role_OSS\/ost1 on ost1...
[ 5870.633988] Lustre: DEBUG MARKER: Mounting /dev/vg_Role_OSS/ost1 on ost1...
[ 5875.608547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 10: e2fsck with mounted filesystem =========== 21:31:49 \(1764883909\)
[ 5875.828319] Lustre: DEBUG MARKER: == mmp test 10: e2fsck with mounted filesystem =========== 21:31:49 (1764883909)
[ 5876.093198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/vg_Role_MDS\/mdt1 on mds1...
[ 5876.311371] Lustre: DEBUG MARKER: Mounting /dev/vg_Role_MDS/mdt1 on mds1...
[ 5881.191840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5881.404865] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5882.983898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/vg_Role_OSS\/ost1 on ost1...
[ 5883.207930] Lustre: DEBUG MARKER: Mounting /dev/vg_Role_OSS/ost1 on ost1...
[ 5889.148437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5889.357995] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5909.901162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disabling MMP on \/dev\/vg_Role_MDS\/mdt1 on mds1 manually...
[ 5910.121354] Lustre: DEBUG MARKER: Disabling MMP on /dev/vg_Role_MDS/mdt1 on mds1 manually...
[ 5922.031959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disabling MMP on \/dev\/vg_Role_OSS\/ost1 on ost1 manually...
[ 5922.253429] Lustre: DEBUG MARKER: Disabling MMP on /dev/vg_Role_OSS/ost1 on ost1 manually...
[ 5934.415962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test complete, duration 447 sec =================== 21:32:48 \(1764883968\)
[ 5934.633552] Lustre: DEBUG MARKER: == mmp test complete, duration 447 sec =================== 21:32:48 (1764883968)
[ 5938.081291] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 5938.475554] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 5944.829194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5945.043995] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5952.662068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5952.880239] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5960.349371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5960.557661] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5968.225237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5968.440040] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5975.845764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5976.064605] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5983.618606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5983.832634] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5991.431664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5991.646166] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5998.971135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5999.188788] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6002.305503] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
[ 6002.717452] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:
[ 6002.942986] Lustre: Mounted lustre-client
[ 6003.129212] Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
[ 6003.540239] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 6005.614708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6005.645162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6005.871718] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6005.875090] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6007.879137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[ 6008.104196] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6008.383452] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[ 6012.166124] Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
[ 6012.518831] Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
[ 6017.989053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === mmp: start cleanup 21:34:11 \(1764884051\) ===
[ 6018.215066] Lustre: DEBUG MARKER: === mmp: start cleanup 21:34:11 (1764884051) ===
[ 6019.640056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === mmp: finish cleanup 21:34:13 \(1764884053\) ===
[ 6019.860113] Lustre: DEBUG MARKER: === mmp: finish cleanup 21:34:13 (1764884053) ===
[ 6020.163892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-dom ============----- Thu Dec  4 09:34:14 PM UTC 2025
[ 6020.390066] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-dom ============----- Thu Dec 4 09:34:14 PM UTC 2025
[ 6025.969033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 
[ 6026.189691] Lustre: DEBUG MARKER: excepting tests:
[ 6026.476062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity-dom: start setup 21:34:20 \(1764884060\) ===
[ 6026.695311] Lustre: DEBUG MARKER: === sanity-dom: start setup 21:34:20 (1764884060) ===
[ 6026.923016] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre2
[ 6027.373823] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre2' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre2
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tc
[ 6027.596378] Lustre: Mounted lustre-client
[ 6027.806513] Lustre: DEBUG MARKER: mount | grep /mnt/lustre2' '
[ 6028.216477] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 6029.695652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6029.711101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6029.935779] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6029.943531] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6030.140838] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 6031.645947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6031.699512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6031.720895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6031.956455] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6032.015829] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6032.024442] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6035.581200] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 6035.979250] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 6043.269941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[ 6043.489662] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6043.757269] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[ 6046.317900] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 6047.840890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6047.843238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6048.084738] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6048.087410] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6048.503049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6048.596659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6048.769026] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6048.834487] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6050.213550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity-dom: finish setup 21:34:44 \(1764884084\) ===
[ 6050.434420] Lustre: DEBUG MARKER: === sanity-dom: finish setup 21:34:44 (1764884084) ===
[ 6050.870945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test 1: write a file on one mount, truncate on the other, write again ========================================================== 21:34:44 \(1764884084\)
[ 6051.093812] Lustre: DEBUG MARKER: == sanity-dom test 1: write a file on one mount, truncate on the other, write again ========================================================== 21:34:44 (1764884084)
[ 6052.390965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test 2: Write with a seek, append, read from a single mountpoint ========================================================== 21:34:46 \(1764884086\)
[ 6052.611688] Lustre: DEBUG MARKER: == sanity-dom test 2: Write with a seek, append, read from a single mountpoint ========================================================== 21:34:46 (1764884086)
[ 6053.910988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test 3: Truncate over DoM size on different nodes ========================================================== 21:34:47 \(1764884087\)
[ 6054.131698] Lustre: DEBUG MARKER: == sanity-dom test 3: Truncate over DoM size on different nodes ========================================================== 21:34:47 (1764884087)
[ 6055.477338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test 4: DoM: glimpse doesn\'t produce duplicated locks ========================================================== 21:34:49 \(1764884089\)
[ 6055.701505] Lustre: DEBUG MARKER: == sanity-dom test 4: DoM: glimpse doesn't produce duplicated locks ========================================================== 21:34:49 (1764884089)
[ 6057.933117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test 5: DoM truncate deadlock ============== 21:34:51 \(1764884091\)
[ 6058.154516] Lustre: DEBUG MARKER: == sanity-dom test 5: DoM truncate deadlock ============== 21:34:51 (1764884091)
[ 6061.524902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test 6: Race two writes, check file size === 21:34:55 \(1764884095\)
[ 6061.745801] Lustre: DEBUG MARKER: == sanity-dom test 6: Race two writes, check file size === 21:34:55 (1764884095)
[ 6064.064393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test 7: Stale pages after read-on-open ===== 21:34:58 \(1764884098\)
[ 6064.282485] Lustre: DEBUG MARKER: == sanity-dom test 7: Stale pages after read-on-open ===== 21:34:58 (1764884098)
[ 6065.617757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test 8: verify the default size of component ========================================================== 21:34:59 \(1764884099\)
[ 6065.839534] Lustre: DEBUG MARKER: == sanity-dom test 8: verify the default size of component ========================================================== 21:34:59 (1764884099)
[ 6067.160289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test fsx: Dual-mount fsx with DoM files ==== 21:35:01 \(1764884101\)
[ 6067.389819] Lustre: DEBUG MARKER: == sanity-dom test fsx: Dual-mount fsx with DoM files ==== 21:35:01 (1764884101)
[ 6075.249747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test sanity: Run sanity with Data-on-MDT files ========================================================== 21:35:09 \(1764884109\)
[ 6075.477098] Lustre: DEBUG MARKER: == sanity-dom test sanity: Run sanity with Data-on-MDT files ========================================================== 21:35:09 (1764884109)
[ 6081.085784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark - need MDS1_VERSION \< v2_14_55-100-g8a84c7f9c7 \(34539520 \< 34486116\) for LU-14927, skip 0f
[ 6081.302742] Lustre: DEBUG MARKER: - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34539520 < 34486116) for LU-14927, skip 0f
[ 6081.581700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 42a 42c 42b 118c 118d 407 119i 817 906
[ 6081.800354] Lustre: DEBUG MARKER: excepting tests: 42a 42c 42b 118c 118d 407 119i 817 906
[ 6082.086903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity: start setup 21:35:16 \(1764884116\) ===
[ 6082.309354] Lustre: DEBUG MARKER: === sanity: start setup 21:35:16 (1764884116) ===
[ 6082.538403] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[ 6082.911703] LustreError: 104188:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91fe98962000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6082.911818] LustreError: 104188:0:(lov_obd.c:780:lov_cleanup()) Skipped 6 previous similar messages
[ 6082.912856] LustreError: 104188:0:(obd_class.h:478:obd_check_dev()) Device 20 not setup
[ 6082.912971] LustreError: 104188:0:(obd_class.h:478:obd_check_dev()) Skipped 17 previous similar messages
[ 6082.932968] Lustre: Unmounted lustre-client
[ 6083.123806] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 6084.677713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6084.710115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6084.748380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6084.997315] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6085.027758] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6085.052698] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6088.560217] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 6088.960724] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 6094.977123] Autotest: Test running for 100 minutes (lustre-master_full-part-1_4681.175)
[ 6096.043138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[ 6096.264440] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6096.532129] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[ 6099.111196] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 6100.647976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6100.653134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6100.897234] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6100.899770] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6101.341556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6101.402852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6101.617195] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6101.675001] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6103.026630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity: finish setup 21:35:37 \(1764884137\) ===
[ 6103.247471] Lustre: DEBUG MARKER: === sanity: finish setup 21:35:37 (1764884137) ===
[ 6104.140811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36a: MDS utime check \(mknod, utime\) ======= 21:35:38 \(1764884138\)
[ 6104.363567] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) ======= 21:35:38 (1764884138)
[ 6105.714957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36b: OST utime check \(open, utime\) ======== 21:35:39 \(1764884139\)
[ 6105.939282] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ======== 21:35:39 (1764884139)
[ 6107.274371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36c: non-root MDS utime check \(mknod, utime\) ========================================================== 21:35:41 \(1764884141\)
[ 6107.501009] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================================== 21:35:41 (1764884141)
[ 6108.855846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36d: non-root OST utime check \(open, utime\) ========================================================== 21:35:42 \(1764884142\)
[ 6109.076774] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) ========================================================== 21:35:42 (1764884142)
[ 6110.399598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36e: utime on non-owned file \(should return error\) ========================================================== 21:35:44 \(1764884144\)
[ 6110.625210] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ========================================================== 21:35:44 (1764884144)
[ 6111.967644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36f: utime on file racing with OST BRW write ==================================================================== 21:35:45 \(1764884145\)
[ 6112.193231] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ==================================================================== 21:35:45 (1764884145)
[ 6114.572510] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36g: FMD cache expiry =============================================================================== 21:35:48 \(1764884148\)
[ 6114.797483] Lustre: DEBUG MARKER: == sanity test 36g: FMD cache expiry =============================================================================== 21:35:48 (1764884148)
[ 6159.122074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36h: utime on file racing with OST BRW write ==================================================================== 21:36:33 \(1764884193\)
[ 6159.346261] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ==================================================================== 21:36:33 (1764884193)
[ 6161.704874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36i: change mtime on striped directory ==== 21:36:35 \(1764884195\)
[ 6161.927473] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ==== 21:36:35 (1764884195)
[ 6162.207975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity test_36i needs \>= 2 MDTs
[ 6162.429493] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs
[ 6162.739186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39a: mtime changed on create ============== 21:36:36 \(1764884196\)
[ 6162.957078] Lustre: DEBUG MARKER: == sanity test 39a: mtime changed on create ============== 21:36:36 (1764884196)
[ 6166.293641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39b: mtime change on open, link, unlink, rename  ================================================================ 21:36:40 \(1764884200\)
[ 6166.518190] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ================================================================ 21:36:40 (1764884200)
[ 6169.008440] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39c: mtime change on rename ===================================================================================== 21:36:43 \(1764884203\)
[ 6169.245408] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename ===================================================================================== 21:36:43 (1764884203)
[ 6172.671904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39d: create, utime, stat ======================================================================================== 21:36:46 \(1764884206\)
[ 6172.897525] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ======================================================================================== 21:36:46 (1764884206)
[ 6174.233187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39e: create, stat, utime, stat ================================================================================== 21:36:48 \(1764884208\)
[ 6174.468661] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ================================================================================== 21:36:48 (1764884208)
[ 6175.884123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39f: create, stat, sleep, utime, stat =========================================================================== 21:36:49 \(1764884209\)
[ 6176.109992] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat =========================================================================== 21:36:49 (1764884209)
[ 6179.494903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39g: write, chmod, stat ========================================================================================= 21:36:53 \(1764884213\)
[ 6179.737643] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat ========================================================================================= 21:36:53 (1764884213)
[ 6183.119939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39h: write, utime within one second, stat ======================================================================= 21:36:57 \(1764884217\)
[ 6183.344805] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ======================================================================= 21:36:57 (1764884217)
[ 6185.748393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39i: write, rename, stat ======================================================================================== 21:36:59 \(1764884219\)
[ 6185.966912] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ======================================================================================== 21:36:59 (1764884219)
[ 6188.335921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39j: write, rename, close, stat ================================================================================= 21:37:02 \(1764884222\)
[ 6188.563497] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ================================================================================= 21:37:02 (1764884222)
[ 6189.155391] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 debug_mb=150
[ 6191.279838] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=11
[ 6191.683906] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=trace+inode+super+iotrace+malloc+cache+info+ioctl+neterror+net+warning+buffs+other+dentry+nettrace+page+dlmtrace+error+emerg+ha+rpctrace+vfstrace+reada+mmap+config+console+quota+sec+lfsck+hsm+snapshot+layout
[ 6193.659542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39k: write, utime, close, stat ================================================================================== 21:37:07 \(1764884227\)
[ 6193.885093] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ================================================================================== 21:37:07 (1764884227)
[ 6196.861629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39l: directory atime update ===================================================================================== 21:37:10 \(1764884230\)
[ 6197.090518] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update ===================================================================================== 21:37:10 (1764884230)
[ 6205.790322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39m: test atime and mtime before 1970 ===== 21:37:19 \(1764884239\)
[ 6206.026528] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ===== 21:37:19 (1764884239)
[ 6209.397892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39n: check that O_NOATIME is honored ====== 21:37:23 \(1764884243\)
[ 6209.621789] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ====== 21:37:23 (1764884243)
[ 6221.978993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39o: directory cached attributes updated after create ========================================================== 21:37:35 \(1764884255\)
[ 6222.206532] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ========================================================== 21:37:35 (1764884255)
[ 6223.545171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39p: remote directory cached attributes updated after create ================================================================== 21:37:37 \(1764884257\)
[ 6223.770219] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ================================================================== 21:37:37 (1764884257)
[ 6224.045161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity test_39p needs \>= 2 MDTs
[ 6224.265757] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs
[ 6224.573548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39r: lazy atime update on OST ============= 21:37:38 \(1764884258\)
[ 6224.800631] Lustre: DEBUG MARKER: == sanity test 39r: lazy atime update on OST ============= 21:37:38 (1764884258)
[ 6239.078997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39q: close won\'t zero out atime =========== 21:37:53 \(1764884273\)
[ 6239.301845] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime =========== 21:37:53 (1764884273)
[ 6240.624075] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39s: relatime is supported ================ 21:37:54 \(1764884274\)
[ 6240.851871] Lustre: DEBUG MARKER: == sanity test 39s: relatime is supported ================ 21:37:54 (1764884274)
[ 6247.908927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39u: stat race ============================ 21:38:01 \(1764884281\)
[ 6248.132624] Lustre: DEBUG MARKER: == sanity test 39u: stat race ============================ 21:38:01 (1764884281)
[ 6263.944658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 40: failed open\(O_TRUNC\) doesn\'t truncate ======================================================================= 21:38:17 \(1764884297\)
[ 6264.167870] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ======================================================================= 21:38:17 (1764884297)
[ 6265.505280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 41: test small file write + fstat =============================================================================== 21:38:19 \(1764884299\)
[ 6265.727566] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat =============================================================================== 21:38:19 (1764884299)
[ 6268.048965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 42d: test complete truncate of file with cached dirty data ========================================================== 21:38:22 \(1764884302\)
[ 6268.274071] Lustre: DEBUG MARKER: == sanity test 42d: test complete truncate of file with cached dirty data ========================================================== 21:38:22 (1764884302)
[ 6269.095965] Lustre: DEBUG MARKER: sync
[ 6269.547419] Lustre: DEBUG MARKER: /usr/bin/lfs df
[ 6269.956126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-*.cur_*grant_bytes
[ 6271.592350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 42e: verify sub-RPC writes are not done synchronously ========================================================== 21:38:25 \(1764884305\)
[ 6271.816305] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ========================================================== 21:38:25 (1764884305)
[ 6272.405323] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=ha
[ 6275.875528] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=trace+inode+super+iotrace+malloc+cache+info+ioctl+neterror+net+warning+buffs+other+dentry+nettrace+page+dlmtrace+error+emerg+ha+rpctrace+vfstrace+reada+mmap+config+console+quota+sec+lfsck+hsm+snapshot+layout
[ 6277.061634] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=ha
[ 6279.450343] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=trace+inode+super+iotrace+malloc+cache+info+ioctl+neterror+net+warning+buffs+other+dentry+nettrace+page+dlmtrace+error+emerg+ha+rpctrace+vfstrace+reada+mmap+config+console+quota+sec+lfsck+hsm+snapshot+layout
[ 6329.919156] Lustre: DEBUG MARKER: sync
[ 6330.406058] Lustre: DEBUG MARKER: /usr/bin/lfs df
[ 6330.817676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-*.cur_*grant_bytes
[ 6332.445343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43A: execution of file opened for write should return -ETXTBSY ========================================================== 21:39:26 \(1764884366\)
[ 6332.673139] Lustre: DEBUG MARKER: == sanity test 43A: execution of file opened for write should return -ETXTBSY ========================================================== 21:39:26 (1764884366)
[ 6335.196002] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43a: open\(RDWR\) of file being executed should return -ETXTBSY ========================================================== 21:39:29 \(1764884369\)
[ 6335.425774] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================================================== 21:39:29 (1764884369)
[ 6337.840343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43b: truncate of file being executed should return -ETXTBSY ========================================================== 21:39:31 \(1764884371\)
[ 6338.070494] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY ========================================================== 21:39:31 (1764884371)
[ 6340.460817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43c: md5sum of copy into lustre =========== 21:39:34 \(1764884374\)
[ 6340.678677] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre =========== 21:39:34 (1764884374)
[ 6342.116508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 46: dirtying a previously written page ========================================================================== 21:39:36 \(1764884376\)
[ 6342.340564] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ========================================================================== 21:39:36 (1764884376)
[ 6343.891728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56r: check lfs find -size works =========== 21:39:37 \(1764884377\)
[ 6344.134318] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works =========== 21:39:37 (1764884377)
[ 6346.221560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101e: check read-ahead for small read\(1k\) for small files\(500k\) ========================================================== 21:39:40 \(1764884380\)
[ 6346.449548] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 21:39:40 (1764884380)
[ 6360.683047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119a: Short directIO read must return actual read amount ========================================================== 21:39:54 \(1764884394\)
[ 6360.913554] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 21:39:54 (1764884394)
[ 6362.442157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131a: test iov\'s crossing stripe boundary for writev\/readv ========================================================== 21:39:56 \(1764884396\)
[ 6362.664087] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 21:39:56 (1764884396)
[ 6364.293903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131b: test append writev ================== 21:39:58 \(1764884398\)
[ 6364.522620] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 21:39:58 (1764884398)
[ 6366.288438] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131c: test read\/write on file w\/o objects ========================================================== 21:40:00 \(1764884400\)
[ 6366.515738] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 21:40:00 (1764884400)
[ 6367.910782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131d: test short read ===================== 21:40:01 \(1764884401\)
[ 6368.141292] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 21:40:01 (1764884401)
[ 6369.579641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131e: test read hitting hole ============== 21:40:03 \(1764884403\)
[ 6369.808306] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 21:40:03 (1764884403)
[ 6371.273353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150a: truncate\/append tests =============== 21:40:05 \(1764884405\)
[ 6371.502084] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 21:40:05 (1764884405)
[ 6377.491701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150b: Verify fallocate \(prealloc\) functionality ========================================================== 21:40:11 \(1764884411\)
[ 6377.720510] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 21:40:11 (1764884411)
[ 6388.133860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 21:40:22 \(1764884422\)
[ 6388.359516] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 21:40:22 (1764884422)
[ 6395.730761] Autotest: Test running for 105 minutes (lustre-master_full-part-1_4681.175)
[ 6403.402514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150c: Verify fallocate Size and Blocks ==== 21:40:37 \(1764884437\)
[ 6403.633563] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 21:40:37 (1764884437)
[ 6413.973338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 21:40:47 \(1764884447\)
[ 6414.203714] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 21:40:47 (1764884447)
[ 6424.452686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150f: Verify fallocate punch functionality ========================================================== 21:40:58 \(1764884458\)
[ 6424.680686] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 21:40:58 (1764884458)
[ 6427.291547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150g: Verify fallocate punch on large range ========================================================== 21:41:01 \(1764884461\)
[ 6427.522996] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 21:41:01 (1764884461)
[ 6439.041934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 21:41:13 \(1764884473\)
[ 6439.269796] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 21:41:13 (1764884473)
[ 6439.570587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity test_150ia need MDS1 version \>= 2.16.50 for falloc zero-range
[ 6439.796915] Lustre: DEBUG MARKER: SKIP: sanity test_150ia need MDS1 version >= 2.16.50 for falloc zero-range
[ 6440.124611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 21:41:14 \(1764884474\)
[ 6440.355171] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 21:41:14 (1764884474)
[ 6440.651237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity test_150ib need MDS1 version \>= 2.16.50 for falloc zero-range
[ 6440.876545] Lustre: DEBUG MARKER: SKIP: sanity test_150ib need MDS1 version >= 2.16.50 for falloc zero-range
[ 6441.208583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150ic: Verify fallocate LARGE zero PREALLOC functionality ========================================================== 21:41:15 \(1764884475\)
[ 6441.438995] Lustre: DEBUG MARKER: == sanity test 150ic: Verify fallocate LARGE zero PREALLOC functionality ========================================================== 21:41:15 (1764884475)
[ 6441.734559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity test_150ic need MDS1 version \>= 2.16.54 for falloc zero-range
[ 6441.962197] Lustre: DEBUG MARKER: SKIP: sanity test_150ic need MDS1 version >= 2.16.54 for falloc zero-range
[ 6442.330858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 21:41:16 \(1764884476\)
[ 6442.560897] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 21:41:16 (1764884476)
[ 6449.675757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 21:41:23 \(1764884483\)
[ 6449.903900] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 21:41:23 (1764884483)
[ 6457.094782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 21:41:31 \(1764884491\)
[ 6457.331030] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 21:41:31 (1764884491)
[ 6464.569154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 21:41:38 \(1764884498\)
[ 6464.800015] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 21:41:38 (1764884498)
[ 6472.246103] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 207a: can refresh layout at glimpse ======= 21:41:46 \(1764884506\)
[ 6472.481600] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 21:41:46 (1764884506)
[ 6473.923213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 207b: can refresh layout at open ========== 21:41:47 \(1764884507\)
[ 6474.152002] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 21:41:47 (1764884507)
[ 6475.909811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 241a: bio vs dio ========================== 21:41:49 \(1764884509\)
[ 6476.140392] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 21:41:49 (1764884509)
[ 6492.466503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 241b: dio vs dio ========================== 21:42:06 \(1764884526\)
[ 6492.700943] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 21:42:06 (1764884526)
[ 6499.488369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 251a: Handling short read and write correctly ========================================================== 21:42:13 \(1764884533\)
[ 6499.721774] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 21:42:13 (1764884533)
[ 6501.951858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 251b: short read restore offset correctly ========================================================== 21:42:15 \(1764884535\)
[ 6502.184805] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 21:42:15 (1764884535)
[ 6509.661473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test complete, duration 434 sec ================ 21:42:23 \(1764884543\)
[ 6509.898445] Lustre: DEBUG MARKER: == sanity test complete, duration 434 sec ================ 21:42:23 (1764884543)
[ 6510.236951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity: start cleanup 21:42:24 \(1764884544\) ===
[ 6510.470538] Lustre: DEBUG MARKER: === sanity: start cleanup 21:42:24 (1764884544) ===
[ 6511.786379] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre2
[ 6512.279777] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre2' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre2
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tc
[ 6512.518158] Lustre: Mounted lustre-client
[ 6512.709450] Lustre: DEBUG MARKER: mount | grep /mnt/lustre2' '
[ 6513.149089] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 6514.641407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug all all
[ 6514.643458] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug all all
[ 6514.870285] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug all all
[ 6514.877228] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug all all
[ 6515.075980] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[ 6515.479302] LustreError: 120261:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91ff828f4800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6515.484566] LustreError: 120261:0:(lov_obd.c:780:lov_cleanup()) Skipped 6 previous similar messages
[ 6515.490444] LustreError: 120261:0:(obd_class.h:478:obd_check_dev()) Device 20 not setup
[ 6515.490573] LustreError: 120261:0:(obd_class.h:478:obd_check_dev()) Skipped 15 previous similar messages
[ 6515.514888] Lustre: Unmounted lustre-client
[ 6516.288268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity: finish cleanup 21:42:30 \(1764884550\) ===
[ 6516.525343] Lustre: DEBUG MARKER: === sanity: finish cleanup 21:42:30 (1764884550) ===
[ 6517.840118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test sanityn: Run sanityn with Data-on-MDT files ========================================================== 21:42:31 \(1764884551\)
[ 6518.061664] Lustre: DEBUG MARKER: == sanity-dom test sanityn: Run sanityn with Data-on-MDT files ========================================================== 21:42:31 (1764884551)
[ 6523.817541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 
[ 6524.038641] Lustre: DEBUG MARKER: excepting tests:
[ 6524.323875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanityn: start setup 21:42:38 \(1764884558\) ===
[ 6524.548140] Lustre: DEBUG MARKER: === sanityn: start setup 21:42:38 (1764884558) ===
[ 6524.785544] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre2
[ 6525.251678] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre2' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre2
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tc
[ 6525.486873] Lustre: Mounted lustre-client
[ 6525.674865] Lustre: DEBUG MARKER: mount | grep /mnt/lustre2' '
[ 6526.089567] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 6527.599192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6527.599983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6527.828276] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6527.831050] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6528.029080] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 6529.570872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6529.608227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6529.623809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6529.908157] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6529.929182] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6529.953798] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6533.619678] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 6534.023974] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 6541.296270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[ 6541.523074] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6541.809926] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[ 6544.417248] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 6545.958701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6545.959530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6546.203915] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6546.206447] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6546.685633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6546.725695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6547.002613] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6547.013495] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6548.434944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanityn: finish setup 21:43:02 \(1764884582\) ===
[ 6548.657930] Lustre: DEBUG MARKER: === sanityn: finish setup 21:43:02 (1764884582) ===
[ 6549.659773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:43:03 \(1764884583\)
[ 6549.882966] Lustre: DEBUG MARKER: == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:43:03 (1764884583)
[ 6551.212156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2a: check cached attribute updates on 2 mtpt\'s ================================================================== 21:43:05 \(1764884585\)
[ 6551.441095] Lustre: DEBUG MARKER: == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 21:43:05 (1764884585)
[ 6552.771385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2b: check cached attribute updates on 2 mtpt\'s ================================================================== 21:43:06 \(1764884586\)
[ 6552.995376] Lustre: DEBUG MARKER: == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 21:43:06 (1764884586)
[ 6554.315207] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2c: check cached attribute updates on 2 mtpt\'s root ============================================================= 21:43:08 \(1764884588\)
[ 6554.537961] Lustre: DEBUG MARKER: == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 21:43:08 (1764884588)
[ 6555.853912] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2d: check cached attribute updates on 2 mtpt\'s root ============================================================= 21:43:09 \(1764884589\)
[ 6556.076086] Lustre: DEBUG MARKER: == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 21:43:09 (1764884589)
[ 6557.360528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:43:11 \(1764884591\)
[ 6557.579880] Lustre: DEBUG MARKER: == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:43:11 (1764884591)
[ 6558.929847] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2f: check attr\/owner updates on DNE with 2 mtpt\'s ========================================================== 21:43:12 \(1764884592\)
[ 6559.153865] Lustre: DEBUG MARKER: == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 21:43:12 (1764884592)
[ 6559.428039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanityn test_2f needs \>= 2 MDTs
[ 6559.656400] Lustre: DEBUG MARKER: SKIP: sanityn test_2f needs >= 2 MDTs
[ 6559.960060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2g: check blocks update on sync write ==== 21:43:13 \(1764884593\)
[ 6560.179628] Lustre: DEBUG MARKER: == sanityn test 2g: check blocks update on sync write ==== 21:43:13 (1764884593)
[ 6561.618867] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:43:15 \(1764884595\)
[ 6561.843181] Lustre: DEBUG MARKER: == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:43:15 (1764884595)
[ 6564.150417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:43:18 \(1764884598\)
[ 6564.373216] Lustre: DEBUG MARKER: == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:43:18 (1764884598)
[ 6565.695093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 6: remove of open file on other node ============================================================================ 21:43:19 \(1764884599\)
[ 6565.918807] Lustre: DEBUG MARKER: == sanityn test 6: remove of open file on other node ============================================================================ 21:43:19 (1764884599)
[ 6567.225984] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 7: remove of open directory on other node ======================================================================= 21:43:21 \(1764884601\)
[ 6567.449118] Lustre: DEBUG MARKER: == sanityn test 7: remove of open directory on other node ======================================================================= 21:43:21 (1764884601)
[ 6568.789473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 8: remove of open special file on other node ==================================================================== 21:43:22 \(1764884602\)
[ 6569.012230] Lustre: DEBUG MARKER: == sanityn test 8: remove of open special file on other node ==================================================================== 21:43:22 (1764884602)
[ 6570.331045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:43:24 \(1764884604\)
[ 6570.557848] Lustre: DEBUG MARKER: == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:43:24 (1764884604)
[ 6571.952092] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 9b: append to striped sparse file ======== 21:43:25 \(1764884605\)
[ 6572.176875] Lustre: DEBUG MARKER: == sanityn test 9b: append to striped sparse file ======== 21:43:25 (1764884605)
[ 6573.645472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:43:27 \(1764884607\)
[ 6573.867922] Lustre: DEBUG MARKER: == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:43:27 (1764884607)
[ 6575.306600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:43:29 \(1764884609\)
[ 6575.532168] Lustre: DEBUG MARKER: == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:43:29 (1764884609)
[ 6576.890522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 11: execution of file opened for write should return error ============================================================== 21:43:30 \(1764884610\)
[ 6577.114463] Lustre: DEBUG MARKER: == sanityn test 11: execution of file opened for write should return error ============================================================== 21:43:30 (1764884610)
[ 6578.441067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 12: test lock ordering \(link, stat, unlink\) ========================================================== 21:43:32 \(1764884612\)
[ 6578.670452] Lustre: DEBUG MARKER: == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 21:43:32 (1764884612)
[ 6697.158543] Autotest: Test running for 110 minutes (lustre-master_full-part-1_4681.175)
[ 6738.410299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:46:12 \(1764884772\)
[ 6738.636228] Lustre: DEBUG MARKER: == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:46:12 (1764884772)
[ 6740.032152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14ab: open\(RDWR\) of executing file returns -ETXTBSY ========================================================== 21:46:14 \(1764884774\)
[ 6740.256259] Lustre: DEBUG MARKER: == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 21:46:14 (1764884774)
[ 6741.650358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:46:15 \(1764884775\)
[ 6741.873651] Lustre: DEBUG MARKER: == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:46:15 (1764884775)
[ 6743.230933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14c: open\(O_TRUNC\) of executing file return -ETXTBSY ============================================================ 21:46:17 \(1764884777\)
[ 6743.456352] Lustre: DEBUG MARKER: == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 21:46:17 (1764884777)
[ 6744.851151] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:46:18 \(1764884778\)
[ 6745.069951] Lustre: DEBUG MARKER: == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:46:18 (1764884778)
[ 6745.383866] Lustre: DEBUG MARKER: /usr/sbin/lctl mark chmod
[ 6745.609431] Lustre: DEBUG MARKER: chmod
[ 6746.981529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 17: resource creation\/LVB creation race ========================================================================= 21:46:21 \(1764884781\)
[ 6747.203460] Lustre: DEBUG MARKER: == sanityn test 17: resource creation/LVB creation race ========================================================================= 21:46:21 (1764884781)
[ 6751.087340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 19: test concurrent uncached read races ========================================================================= 21:46:25 \(1764884785\)
[ 6751.313655] Lustre: DEBUG MARKER: == sanityn test 19: test concurrent uncached read races ========================================================================= 21:46:25 (1764884785)
[ 6754.468558] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  loop 5
[ 6754.704975] Lustre: DEBUG MARKER: loop 5
[ 6755.510828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  loop 10
[ 6755.733816] Lustre: DEBUG MARKER: loop 10
[ 6756.544814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  loop 15
[ 6756.767531] Lustre: DEBUG MARKER: loop 15
[ 6757.578179] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  loop 20
[ 6757.803163] Lustre: DEBUG MARKER: loop 20
[ 6759.594343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 20: test extra readahead page left in cache ============================================================== 21:46:33 \(1764884793\)
[ 6759.818660] Lustre: DEBUG MARKER: == sanityn test 20: test extra readahead page left in cache ============================================================== 21:46:33 (1764884793)
[ 6761.177535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 23:  others should see updated atime while another read============================================================== 21:46:35 \(1764884795\)
[ 6761.400451] Lustre: DEBUG MARKER: == sanityn test 23: others should see updated atime while another read============================================================== 21:46:35 (1764884795)
[ 6824.084786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 27: align non-overlapping extent locks from request ============================================================= 21:47:38 \(1764884858\)
[ 6824.308299] Lustre: DEBUG MARKER: == sanityn test 27: align non-overlapping extent locks from request ============================================================= 21:47:38 (1764884858)
[ 6825.104256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark dd 1 started
[ 6825.330817] Lustre: DEBUG MARKER: dd 1 started
[ 6825.657145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark dd 2 started
[ 6825.877794] Lustre: DEBUG MARKER: dd 2 started
[ 6827.164894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark dd 3 finished
[ 6827.388509] Lustre: DEBUG MARKER: dd 3 finished
[ 6828.723706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 39a: file mtime does not change after rename ========================================================== 21:47:42 \(1764884862\)
[ 6828.948742] Lustre: DEBUG MARKER: == sanityn test 39a: file mtime does not change after rename ========================================================== 21:47:42 (1764884862)
[ 6829.161527] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39a.sanityn
[ 6829.562141] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39a.sanityn
[ 6829.934922] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39a.sanityn-1
[ 6830.299995] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39a.sanityn-1
[ 6831.786354] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 39b: file mtime the same on clients with\/out lock ========================================================== 21:47:45 \(1764884865\)
[ 6832.010583] Lustre: DEBUG MARKER: == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 21:47:45 (1764884865)
[ 6832.202835] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39b.sanityn
[ 6833.574559] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39b.sanityn
[ 6833.946119] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39b.sanityn
[ 6835.464823] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 39c: check truncate mtime update ================================================================================ 21:47:49 \(1764884869\)
[ 6835.687020] Lustre: DEBUG MARKER: == sanityn test 39c: check truncate mtime update ================================================================================ 21:47:49 (1764884869)
[ 6835.876941] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39c.sanityn
[ 6837.247731] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39c.sanityn
[ 6837.615861] Lustre: DEBUG MARKER: stat -c %Y /mnt/lustre/f39c.sanityn
[ 6839.113598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 39d: sync write should update mtime ====== 21:47:53 \(1764884873\)
[ 6839.336102] Lustre: DEBUG MARKER: == sanityn test 39d: sync write should update mtime ====== 21:47:53 (1764884873)
[ 6840.828784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 51a: layout lock: refresh layout should work ========================================================== 21:47:54 \(1764884874\)
[ 6841.052720] Lustre: DEBUG MARKER: == sanityn test 51a: layout lock: refresh layout should work ========================================================== 21:47:54 (1764884874)
[ 6844.426819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 21:47:58 \(1764884878\)
[ 6844.655004] Lustre: DEBUG MARKER: == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 21:47:58 (1764884878)
[ 6850.544409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 21:48:04 \(1764884884\)
[ 6850.767698] Lustre: DEBUG MARKER: == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 21:48:04 (1764884884)
[ 6853.323951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 107a: Basic grouplock conflict =========== 21:48:07 \(1764884887\)
[ 6853.550590] Lustre: DEBUG MARKER: == sanityn test 107a: Basic grouplock conflict =========== 21:48:07 (1764884887)
[ 6857.146813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 21:48:11 \(1764884891\)
[ 6857.370588] Lustre: DEBUG MARKER: == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 21:48:11 (1764884891)
[ 6865.034894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cleanup: ======================================================
[ 6865.258831] Lustre: DEBUG MARKER: cleanup: ======================================================
[ 6865.585160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test complete, duration 347 sec =============== 21:48:19 \(1764884899\)
[ 6865.807932] Lustre: DEBUG MARKER: == sanityn test complete, duration 347 sec =============== 21:48:19 (1764884899)
[ 6866.110891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanityn: start cleanup 21:48:20 \(1764884900\) ===
[ 6866.338522] Lustre: DEBUG MARKER: === sanityn: start cleanup 21:48:20 (1764884900) ===
[ 6867.387009] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[ 6882.893090] LustreError: 132029:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91fe98963800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6882.893247] LustreError: 132029:0:(lov_obd.c:780:lov_cleanup()) Skipped 6 previous similar messages
[ 6882.894124] LustreError: 132029:0:(obd_class.h:478:obd_check_dev()) Device 20 not setup
[ 6882.894223] LustreError: 132029:0:(obd_class.h:478:obd_check_dev()) Skipped 15 previous similar messages
[ 6882.912908] Lustre: Unmounted lustre-client
[ 6883.606710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanityn: finish cleanup 21:48:37 \(1764884917\) ===
[ 6883.840344] Lustre: DEBUG MARKER: === sanityn: finish cleanup 21:48:37 (1764884917) ===
[ 6885.154468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-dom test complete, duration 865 sec ============ 21:48:39 \(1764884919\)
[ 6885.386916] Lustre: DEBUG MARKER: == sanity-dom test complete, duration 865 sec ============ 21:48:39 (1764884919)
[ 6885.673141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity-dom: start cleanup 21:48:39 \(1764884919\) ===
[ 6885.903108] Lustre: DEBUG MARKER: === sanity-dom: start cleanup 21:48:39 (1764884919) ===
[ 6886.845771] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[ 6887.732164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === sanity-dom: finish cleanup 21:48:41 \(1764884921\) ===
[ 6887.955765] Lustre: DEBUG MARKER: === sanity-dom: finish cleanup 21:48:41 (1764884921) ===
[ 6888.277487] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: mds-survey ============----- Thu Dec  4 09:48:42 PM UTC 2025
[ 6888.509242] Lustre: DEBUG MARKER: -----============= acceptance-small: mds-survey ============----- Thu Dec 4 09:48:42 PM UTC 2025
[ 6894.282304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 
[ 6894.502758] Lustre: DEBUG MARKER: excepting tests:
[ 6894.806243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === mds-survey: start setup 21:48:48 \(1764884928\) ===
[ 6895.024533] Lustre: DEBUG MARKER: === mds-survey: start setup 21:48:48 (1764884928) ===
[ 6895.255636] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 6896.772004] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6896.825837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6896.845744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 6897.119937] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6897.132095] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6897.167086] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 6900.882061] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 6901.280938] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 6908.621062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[ 6908.844373] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6909.126447] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[ 6911.753659] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 6913.281328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6913.285860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6913.532587] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6913.537793] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6913.983121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6914.100819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6914.222956] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6914.404110] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6915.819994] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === mds-survey: finish setup 21:49:09 \(1764884949\) ===
[ 6916.046984] Lustre: DEBUG MARKER: === mds-survey: finish setup 21:49:09 (1764884949) ===
[ 6916.291915] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre opts:;
lsof /mnt/lustre || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre | sort -u);
	if [ -n "
[ 6916.656416] LustreError: 134588:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91ff828f9000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6916.656587] LustreError: 134588:0:(lov_obd.c:780:lov_cleanup()) Skipped 6 previous similar messages
[ 6916.657353] LustreError: 134588:0:(obd_class.h:478:obd_check_dev()) Device 10 not setup
[ 6916.657485] LustreError: 134588:0:(obd_class.h:478:obd_check_dev()) Skipped 15 previous similar messages
[ 6916.677542] Lustre: Unmounted lustre-client
[ 6916.868574] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[ 6917.369532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test 1: Metadata survey with zero-stripe === 21:49:11 \(1764884951\)
[ 6917.593572] Lustre: DEBUG MARKER: == mds-survey test 1: Metadata survey with zero-stripe === 21:49:11 (1764884951)
[ 6938.601294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test 2: Metadata survey with stripe_count = 1 ========================================================== 21:49:32 \(1764884972\)
[ 6938.821160] Lustre: DEBUG MARKER: == mds-survey test 2: Metadata survey with stripe_count = 1 ========================================================== 21:49:32 (1764884972)
[ 6939.092070] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: mds-survey test_2 need local client mount
[ 6939.308541] Lustre: DEBUG MARKER: SKIP: mds-survey test_2 need local client mount
[ 6939.508987] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
[ 6939.972434] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:
[ 6940.197230] Lustre: Mounted lustre-client
[ 6940.383361] Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
[ 6940.795086] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 6942.271536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6942.278449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6942.504070] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6942.506721] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6942.800216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test complete, duration 54 sec ============= 21:49:36 \(1764884976\)
[ 6943.027849] Lustre: DEBUG MARKER: == mds-survey test complete, duration 54 sec ============= 21:49:36 (1764884976)
[ 6943.329348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === mds-survey: start cleanup 21:49:37 \(1764884977\) ===
[ 6943.552898] Lustre: DEBUG MARKER: === mds-survey: start cleanup 21:49:37 (1764884977) ===
[ 6945.046349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === mds-survey: finish cleanup 21:49:39 \(1764884979\) ===
[ 6945.264780] Lustre: DEBUG MARKER: === mds-survey: finish cleanup 21:49:39 (1764884979) ===
[ 6945.583451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-lfsck ============----- Thu Dec  4 09:49:39 PM UTC 2025
[ 6945.804594] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Thu Dec 4 09:49:39 PM UTC 2025
[ 6951.553200] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 
[ 6951.771917] Lustre: DEBUG MARKER: excepting tests:
[ 6952.446596] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lfsck  It is NOT necessary to test lfsck under interoperation mode
[ 6952.677538] Lustre: DEBUG MARKER: SKIP: sanity-lfsck It is NOT necessary to test lfsck under interoperation mode
[ 6952.985629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-lnet ============----- Thu Dec  4 09:49:47 PM UTC 2025
[ 6953.207483] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Dec 4 09:49:47 PM UTC 2025
[ 6958.289131] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre opts:;
lsof /mnt/lustre || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre | sort -u);
	if [ -n "
[ 6958.642514] LustreError: 136423:0:(lov_obd.c:780:lov_cleanup()) lustre-clilov-ffff91ff80ac8800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6958.642604] LustreError: 136423:0:(lov_obd.c:780:lov_cleanup()) Skipped 6 previous similar messages
[ 6958.646995] LustreError: 136423:0:(obd_class.h:478:obd_check_dev()) Device 10 not setup
[ 6958.647116] LustreError: 136423:0:(obd_class.h:478:obd_check_dev()) Skipped 17 previous similar messages
[ 6958.667782] Lustre: Unmounted lustre-client
[ 6958.853375] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:;
lsof /mnt/lustre2 || need_kill=no;
if [ x != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	if [ 
[ 6997.761498] Autotest: Test running for 115 minutes (lustre-master_full-part-1_4681.175)
[ 6998.247126] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre opts:-f;
lsof /mnt/lustre || need_kill=no;
if [ x-f != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre | sort -u);
	if [ 
[ 6998.641633] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
if [ $running -ne 0 ] ; then
echo Stopping client $(hostname) /mnt/lustre2 opts:-f;
lsof /mnt/lustre2 || need_kill=no;
if [ x-f != x -a x$need_kill != xno ]; then
	pids=$(lsof -t /mnt/lustre2 | sort -u);
	i
[ 7011.999213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 253 254 410
[ 7012.225995] Lustre: DEBUG MARKER: excepting tests: 253 254 410
[ 7012.875402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:50:46 \(1764885046\)
[ 7013.104087] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:50:46 (1764885046)
[ 7015.383045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 1: Add peer with single nid \(tcp\) ==== 21:50:49 \(1764885049\)
[ 7015.620171] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:50:49 (1764885049)
[ 7017.035854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 2: Add peer with single nid \(o2ib\) === 21:50:51 \(1764885051\)
[ 7017.259289] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:50:51 (1764885051)
[ 7018.655858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:50:52 \(1764885052\)
[ 7018.877227] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:50:52 (1764885052)
[ 7020.262913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 4: Add peer with nidrange \(tcp\) ====== 21:50:54 \(1764885054\)
[ 7020.478669] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:50:54 (1764885054)
[ 7021.960859] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 5: Add peer with nidrange \(o2ib\) ===== 21:50:56 \(1764885056\)
[ 7022.182199] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:50:56 (1764885056)
[ 7023.741922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 6: Add peer with multiple nidranges == 21:50:57 \(1764885057\)
[ 7023.963952] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:50:57 (1764885057)
[ 7025.349545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 7: Various peer delete tests ========= 21:50:59 \(1764885059\)
[ 7025.570933] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:50:59 (1764885059)
[ 7027.903166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 8: Delete single secondary nid from peer \(tcp\) ========================================================== 21:51:01 \(1764885061\)
[ 7028.124557] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:51:01 (1764885061)
[ 7029.496682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 9: Delete all secondary nids from peer \(tcp\) ========================================================== 21:51:03 \(1764885063\)
[ 7029.716821] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:51:03 (1764885063)
[ 7031.123761] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 10: Delete single secondary nid from peer \(o2ib\) ========================================================== 21:51:05 \(1764885065\)
[ 7031.345628] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:51:05 (1764885065)
[ 7032.748548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 11: Delete all secondary nids from peer \(o2ib\) ========================================================== 21:51:06 \(1764885066\)
[ 7032.969582] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:51:06 (1764885066)
[ 7034.347702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 12: Delete a secondary nid from peer \(tcp and o2ib\) ========================================================== 21:51:08 \(1764885068\)
[ 7034.567880] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:51:08 (1764885068)
[ 7035.966955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 13: Delete all secondary nids from peer \(tcp and o2ib\) ========================================================== 21:51:10 \(1764885070\)
[ 7036.206187] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:51:10 (1764885070)
[ 7037.585695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 14: import peer create\/delete with single nid ========================================================== 21:51:11 \(1764885071\)
[ 7037.808602] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:51:11 (1764885071)
[ 7039.298554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 15: import multi peer create\/delete with single nid per peer ========================================================== 21:51:13 \(1764885073\)
[ 7039.518102] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:51:13 (1764885073)
[ 7040.993164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 16: import peer create\/delete with multiple nids ========================================================== 21:51:15 \(1764885075\)
[ 7041.221774] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:51:15 (1764885075)
[ 7042.672271] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 17: import multi peer create\/delete with multiple nids ========================================================== 21:51:16 \(1764885076\)
[ 7042.895436] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:51:16 (1764885076)
[ 7044.453418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:51:18 \(1764885078\)
[ 7044.672643] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:51:18 (1764885078)
[ 7046.100944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:51:20 \(1764885080\)
[ 7046.323557] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:51:20 (1764885080)
[ 7047.820709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 19: Add peer with single nid \(gni\) === 21:51:21 \(1764885081\)
[ 7048.042931] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:51:21 (1764885081)
[ 7049.472959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:51:23 \(1764885083\)
[ 7049.697524] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:51:23 (1764885083)
[ 7051.088548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 21: Add peer with nidrange \(gni\) ===== 21:51:25 \(1764885085\)
[ 7051.308293] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:51:25 (1764885085)
[ 7052.768170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 22: Delete single secondary nid from peer \(gni\) ========================================================== 21:51:26 \(1764885086\)
[ 7052.984758] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:51:26 (1764885086)
[ 7054.395342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 23: Delete all secondary nids from peer \(gni\) ========================================================== 21:51:28 \(1764885088\)
[ 7054.632187] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:51:28 (1764885088)
[ 7056.053926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 24: Delete a secondary nid from peer \(tcp, o2ib and gni\) ========================================================== 21:51:30 \(1764885090\)
[ 7056.273736] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:51:30 (1764885090)
[ 7057.661552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 25: Delete all secondary nids from peer \(tcp, gni and o2ib\) ========================================================== 21:51:31 \(1764885091\)
[ 7057.883355] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:51:31 (1764885091)
[ 7059.276656] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:51:33 \(1764885093\)
[ 7059.496398] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:51:33 (1764885093)
[ 7060.836387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:51:34 \(1764885094\)
[ 7061.059377] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:51:34 (1764885094)
[ 7062.431614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 28: Test peer_list =================== 21:51:36 \(1764885096\)
[ 7062.651798] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:51:36 (1764885096)
[ 7064.003630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 50: Enable\/disable routing via yaml import ========================================================== 21:51:38 \(1764885098\)
[ 7064.224180] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:51:38 (1764885098)
[ 7074.333547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 51: Set numa range via yaml import === 21:51:48 \(1764885108\)
[ 7074.553754] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:51:48 (1764885108)
[ 7075.973506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:51:50 \(1764885110\)
[ 7076.198085] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:51:50 (1764885110)
[ 7077.709748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:51:51 \(1764885111\)
[ 7077.932112] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:51:51 (1764885111)
[ 7079.376823] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 100: Add route with single gw ======== 21:51:53 \(1764885113\)
[ 7079.601223] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:51:53 (1764885113)
[ 7085.599189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7085.731381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7085.846712] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7086.025809] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7089.394950] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7089.598900] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7092.675202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7092.902685] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7096.317297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7096.598081] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7098.744557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7098.764196] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7099.016228] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7099.063524] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7102.643494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7102.854882] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7106.027605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7106.238319] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7109.459072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7109.686051] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7114.135339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 101: Add route with multiple gw ====== 21:52:28 \(1764885148\)
[ 7114.360752] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:52:28 (1764885148)
[ 7116.779644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 102: Delete route with single gw ===== 21:52:30 \(1764885150\)
[ 7117.004230] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:52:30 (1764885150)
[ 7121.417130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7121.449410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7121.679817] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7121.742066] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7125.101714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7125.311516] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7128.278322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7128.497783] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7131.859968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7132.100362] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7134.278307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7134.376180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7134.555766] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7134.636510] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7138.193367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7138.406656] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7141.585690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7141.796458] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7145.040335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7145.248274] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7149.447142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 103: Delete route with multiple gw === 21:53:03 \(1764885183\)
[ 7149.668536] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:53:03 (1764885183)
[ 7152.046126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 104: Set\/check response_tracking param ========================================================== 21:53:06 \(1764885186\)
[ 7152.263245] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:53:06 (1764885186)
[ 7157.091766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:53:11 \(1764885191\)
[ 7157.322323] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:53:11 (1764885191)
[ 7161.720901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7161.781492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7162.020604] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7162.059375] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7165.434526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7165.645193] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7168.616105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7168.829379] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7172.194033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7172.429312] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7174.570565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7174.584449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7174.850847] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7174.900196] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7178.457700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7178.669633] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7181.813151] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7182.022159] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7185.241271] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7185.448682] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7189.516920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 106: Deleting GW peer should fail ==== 21:53:43 \(1764885223\)
[ 7189.744189] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:53:43 (1764885223)
[ 7194.101390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7194.159136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7194.371647] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7194.434840] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7197.791304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7198.001285] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7200.960891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7201.182078] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7204.522961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7204.779389] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7206.880747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7206.935865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7207.162646] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7207.198825] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7210.759244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7210.976480] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7214.194410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7214.405989] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7217.610186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7217.832588] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7221.901178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 107: Deleting extra interface doesn\'t crash node ========================================================== 21:54:15 \(1764885255\)
[ 7222.120126] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:54:15 (1764885255)
[ 7227.569217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 108: Check Multi-Rail setup ========== 21:54:21 \(1764885261\)
[ 7227.792621] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:54:21 (1764885261)
[ 7233.309559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 109: Add NI using a network interface alias \(LU-16859\) ========================================================== 21:54:27 \(1764885267\)
[ 7233.533026] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:54:27 (1764885267)
[ 7239.504165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:54:33 \(1764885273\)
[ 7239.731821] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:54:33 (1764885273)
[ 7242.591355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 111: Test many routes ================ 21:54:36 \(1764885276\)
[ 7242.813454] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:54:36 (1764885276)
[ 7247.206179] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7247.275359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7247.451253] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7247.540882] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7250.885463] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7251.099496] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7254.061905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7254.280351] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7257.622500] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7257.895299] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7259.947341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7260.116325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7260.175847] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7260.391962] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7263.920054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7264.138521] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7267.295242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7267.507184] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7270.699952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7270.915873] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7278.795798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 112: multiple net configurations ===== 21:55:12 \(1764885312\)
[ 7279.022281] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:55:12 (1764885312)
[ 7284.521105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:55:18 \(1764885318\)
[ 7284.744180] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:55:18 (1764885318)
[ 7287.208955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:55:21 \(1764885321\)
[ 7287.434317] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:55:21 (1764885321)
[ 7293.191650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:55:27 \(1764885327\)
[ 7293.418305] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:55:27 (1764885327)
[ 7298.721243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:55:32 \(1764885332\)
[ 7298.942577] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:55:32 (1764885332)
[ 7299.217510] Autotest: Test running for 120 minutes (lustre-master_full-part-1_4681.175)
[ 7301.845516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:55:35 \(1764885335\)
[ 7302.069918] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:55:35 (1764885335)
[ 7307.044252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:55:41 \(1764885341\)
[ 7307.270184] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:55:41 (1764885341)
[ 7308.668469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:55:42 \(1764885342\)
[ 7308.892505] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:55:42 (1764885342)
[ 7311.788073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:55:45 \(1764885345\)
[ 7312.009833] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:55:45 (1764885345)
[ 7313.379769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:55:47 \(1764885347\)
[ 7313.618980] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:55:47 (1764885347)
[ 7315.273910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 159: Check import of ip2nets with IPv6 \/128 CIDR notation ========================================================== 21:55:49 \(1764885349\)
[ 7315.495395] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:55:49 (1764885349)
[ 7317.156293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:55:51 \(1764885351\)
[ 7317.378123] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:55:51 (1764885351)
[ 7323.257308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:55:57 \(1764885357\)
[ 7323.481165] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:55:57 (1764885357)
[ 7325.483258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:55:59 \(1764885359\)
[ 7325.704355] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:55:59 (1764885359)
[ 7327.092561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 199: load lnet w\/o module option, configure interface twice ========================================================== 21:56:01 \(1764885361\)
[ 7327.314604] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:56:01 (1764885361)
[ 7328.815350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 200: load lnet w\/o module option, configure in a non-default namespace ========================================================== 21:56:02 \(1764885362\)
[ 7329.034531] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:56:02 (1764885362)
[ 7333.772001] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:56:07 \(1764885367\)
[ 7333.989296] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:56:07 (1764885367)
[ 7338.688311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:56:12 \(1764885372\)
[ 7338.909244] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:56:12 (1764885372)
[ 7343.601639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:56:17 \(1764885377\)
[ 7343.828165] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:56:17 (1764885377)
[ 7348.881824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:56:22 \(1764885382\)
[ 7349.103980] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:56:22 (1764885382)
[ 7354.529319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7354.732850] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7360.463946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7360.680209] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7362.323777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:56:36 \(1764885396\)
[ 7362.549490] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:56:36 (1764885396)
[ 7367.880956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7368.092419] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7371.734621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7371.942864] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7375.360797] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7375.579263] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7380.895296] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0
[ 7381.108970] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0
[ 7386.629263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7386.844125] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7388.536561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:57:02 \(1764885422\)
[ 7388.766960] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:57:02 (1764885422)
[ 7394.164324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7394.379158] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7399.370756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7399.580136] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7401.312703] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:57:15 \(1764885435\)
[ 7401.535593] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:57:15 (1764885435)
[ 7406.923375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7407.145770] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7410.749677] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7410.962318] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7414.391139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7414.604831] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7418.931873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0
[ 7419.143111] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0
[ 7423.627123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7423.839094] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7425.458933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:57:39 \(1764885459\)
[ 7425.677994] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:57:39 (1764885459)
[ 7435.704057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:57:49 \(1764885469\)
[ 7435.924817] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:57:49 (1764885469)
[ 7441.326437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7441.538745] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7445.941385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7446.151492] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7451.769848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7451.986358] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7455.598077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7455.808851] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7459.282645] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7459.503917] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7463.167409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0
[ 7463.387426] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0
[ 7467.846667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7468.058056] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7469.737770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 210: Local NI recovery checks ======== 21:58:23 \(1764885503\)
[ 7469.957095] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:58:23 (1764885503)
[ 7490.474017] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 211: Remote NI recovery checks ======= 21:58:44 \(1764885524\)
[ 7490.693953] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:58:44 (1764885524)
[ 7526.719345] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 212: Check discovery refcount loss bug \(LU-14627\) ========================================================== 21:59:20 \(1764885560\)
[ 7526.942644] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:59:20 (1764885560)
[ 7532.405503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7532.623567] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7533.570288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
[ 7533.794509] Lustre: DEBUG MARKER: Initial discovery
[ 7534.371185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
[ 7534.600327] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
[ 7534.919512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-42vm5.trevis.whamcloud.com to discover 10.240.38.81@tcp \(in background\)
[ 7535.143176] Lustre: DEBUG MARKER: Force trevis-42vm5.trevis.whamcloud.com to discover 10.240.38.81@tcp (in background)
[ 7535.422708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-24vm2 to discover 10.240.39.188@tcp
[ 7535.641062] Lustre: DEBUG MARKER: Force trevis-24vm2 to discover 10.240.39.188@tcp
[ 7536.217898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 469696
[ 7536.440444] Lustre: DEBUG MARKER: Wait for 469696
[ 7538.482161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 469696
[ 7538.697475] Lustre: DEBUG MARKER: Finished wait on 469696
[ 7542.197817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7542.409035] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing unload_modules_local
[ 7544.083184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:59:38 \(1764885578\)
[ 7544.314561] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:59:38 (1764885578)
[ 7549.490174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:59:43 \(1764885583\)
[ 7549.715818] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:59:43 (1764885583)
[ 7552.750619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:59:46 \(1764885586\)
[ 7552.973326] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:59:46 (1764885586)
[ 7557.685513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:59:51 \(1764885591\)
[ 7557.910759] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:59:51 (1764885591)
[ 7560.740323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 217: Don\'t leak memory when discovering peer with nnis \<= 1 ========================================================== 21:59:54 \(1764885594\)
[ 7560.963175] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:59:54 (1764885594)
[ 7563.828690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:59:57 \(1764885597\)
[ 7564.051330] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:59:57 (1764885597)
[ 7565.185483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_218 Need two LNet interfaces
[ 7565.408256] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces
[ 7565.726824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 219: Consolidate peer entries ======== 21:59:59 \(1764885599\)
[ 7565.950320] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:59:59 (1764885599)
[ 7567.650722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 220: Add routes w\/default options - check aliveness ========================================================== 22:00:01 \(1764885601\)
[ 7567.878264] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:00:01 (1764885601)
[ 7572.324266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7572.452760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7572.563181] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7572.708406] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7576.097480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7576.316054] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7579.414041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7579.630490] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7583.032985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7583.261892] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7585.482951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7585.516873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7585.726507] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7585.839101] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7589.435162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7589.645663] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7592.824207] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7593.042671] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7596.382824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7596.604814] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7601.324528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
[ 7601.564708] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
[ 7602.051330] Autotest: Test running for 125 minutes (lustre-master_full-part-1_4681.175)
[ 7602.842722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
[ 7603.060292] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
[ 7643.654153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 221: Add routes w\/DD disabled - check aliveness ========================================================== 22:01:17 \(1764885677\)
[ 7643.877514] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:01:17 (1764885677)
[ 7648.295961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7648.462328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7648.533710] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7648.755553] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7652.158208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7652.371105] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7655.461533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7655.675897] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7659.041321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7659.272001] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7661.483482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7661.570333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7661.768144] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7661.859013] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7665.457096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7665.669856] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7668.881097] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7669.089811] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7672.475747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7672.690279] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7679.000904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:01:53 \(1764885713\)
[ 7679.224136] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:01:53 (1764885713)
[ 7683.667611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7683.834189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7683.908166] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7684.089427] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7687.463942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7687.668140] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7690.755985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7690.971657] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7694.375677] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7694.611275] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7696.789721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7696.923056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7697.043394] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7697.208152] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7700.785267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7701.004838] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7704.172687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7704.378257] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7707.687169] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7707.910502] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7738.202489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 223: Check avoid_asym_router_failure=1 w\/DD disabled ========================================================== 22:02:52 \(1764885772\)
[ 7738.429062] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:02:52 (1764885772)
[ 7742.886755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7742.998611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7743.153739] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7743.249840] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7746.674297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7746.885237] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7749.976949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7750.197045] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7753.543666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7753.811584] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7756.050356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7756.076742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7756.315064] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7756.379303] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7759.975505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7760.185836] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7763.370019] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7763.579651] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7766.897489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7767.118905] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7797.942832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:03:51 \(1764885831\)
[ 7798.167444] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:03:51 (1764885831)
[ 7802.626227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7802.797184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7802.858313] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7803.091724] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7806.465700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7806.676838] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7809.800543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7810.021427] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7813.386763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7813.635624] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7815.841052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7815.853902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7816.145136] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7816.148220] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7819.789984] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7820.006106] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7823.195305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7823.408675] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7826.774516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7826.990480] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7851.431124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 225: Check avoid_asym_router_failure=0 w\/DD disabled ========================================================== 22:04:45 \(1764885885\)
[ 7851.651068] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:04:45 (1764885885)
[ 7856.057333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7856.216599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7856.288876] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7856.463705] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7859.839884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7860.050760] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7863.122854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7863.336163] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7866.729886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7866.987003] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7869.053511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7869.229332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7869.285574] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7869.524757] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7873.094692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7873.306933] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7876.474965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7876.686763] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7880.026660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7880.240736] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7902.329880] Autotest: Test running for 130 minutes (lustre-master_full-part-1_4681.175)
[ 7904.790436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:05:38 \(1764885938\)
[ 7905.012241] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:05:38 (1764885938)
[ 7905.287532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_226 need MDS \>= 2.15.62.31 for refcnt fix LU-17440
[ 7905.517967] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 need MDS >= 2.15.62.31 for refcnt fix LU-17440
[ 7905.824256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 227: Check router peer health w\/DD disabled ========================================================== 22:05:39 \(1764885939\)
[ 7906.047805] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:05:39 (1764885939)
[ 7906.317491] Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod
[ 7906.848671] LNet: 164228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
[ 7906.850669] LNetError: 164228:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found
[ 7906.855720] LNet: Removed LNI 10.240.41.196@tcp
[ 7906.975418] Key type .llcrypt unregistered
[ 7906.975560] Key type ._llcrypt unregistered
[ 7908.877721] Key type ._llcrypt registered
[ 7908.878713] Key type .llcrypt registered
[ 7909.321580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 7909.338828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 7909.566545] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 7909.573331] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 7909.646645] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
[ 7909.649093] alg: No test for adler32 (adler32-zlib)
[ 7911.192245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7911.346118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7911.431622] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7911.599701] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7911.888819] Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all
[ 7912.099165] LNet: Added LNI 10.240.41.196@tcp [8/256/0/180]
[ 7915.067546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7915.272940] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7918.377501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7918.596758] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7920.099416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7920.099832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7920.326191] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7920.331560] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7921.247447] Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod
[ 7921.475570] LNet: 165804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
[ 7921.476785] LNetError: 165804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found
[ 7921.480810] LNet: Removed LNI 10.240.41.196@tcp
[ 7921.554738] Key type .llcrypt unregistered
[ 7921.555619] Key type ._llcrypt unregistered
[ 7923.720729] Key type ._llcrypt registered
[ 7923.721704] Key type .llcrypt registered
[ 7923.965060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers
[ 7924.032407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers
[ 7924.074732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers
[ 7924.260552] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffe
[ 7924.361923] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffe
[ 7924.379029] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffe
[ 7924.442968] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
[ 7924.446310] alg: No test for adler32 (adler32-zlib)
[ 7926.445424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers
[ 7926.483177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers
[ 7926.737812] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffe
[ 7926.777190] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffe
[ 7927.105482] Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure 
[ 7930.407951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7930.623975] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7933.795685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7934.006667] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7934.291511] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 7935.764574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7935.773215] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7936.015535] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7936.016442] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7936.026357] LNet: Added LNI 10.240.41.196@tcp1 [8/256/0/180]
[ 7939.724795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7939.949099] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 7940.668063] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids
[ 7941.705894] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp --gateway 10.240.38.81@tcp1
[ 7941.886910] LNet: 167964:0:(router.c:718:lnet_add_route()) Consider turning discovery on to enable full Multi-Rail routing functionality
[ 7941.887011] LNet: 167964:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled
[ 7943.523090] Lustre: DEBUG MARKER: /usr/sbin/lctl show_route
[ 7943.884619] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route show -v
[ 7944.245341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n routes
[ 7944.948316] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping 10.240.39.188@tcp
[ 7968.736673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity-lnet test_227: @@@@@@ FAIL: Expected dropped \> 0 found 0 
[ 7968.959729] Lustre: DEBUG MARKER: sanity-lnet test_227: @@@@@@ FAIL: Expected dropped > 0 found 0
[ 7969.236537] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_4681_175_3e4a9bbf-3bb6-4eb4-addf-bb27972412b0//sanity-lnet.test_227.debug_log.$(hostname -s).1764886003.log;
		dmesg > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_
[ 7970.224768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:06:44 \(1764886004\)
[ 7970.445927] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:06:44 (1764886004)
[ 7970.711557] Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod
[ 7970.931218] LNet: 168607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
[ 7970.933278] LNetError: 168607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found
[ 7970.940556] LNet: Removed LNI 10.240.41.196@tcp1
[ 7971.015521] Key type .llcrypt unregistered
[ 7971.016418] Key type ._llcrypt unregistered
[ 7973.477469] Key type ._llcrypt registered
[ 7973.478433] Key type .llcrypt registered
[ 7973.741956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 7973.743330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 7974.006653] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 7974.014228] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 7974.083724] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
[ 7974.086247] alg: No test for adler32 (adler32-zlib)
[ 7976.172193] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7976.285587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7976.449290] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 7976.560069] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 7976.832675] Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all
[ 7977.052357] LNet: Added LNI 10.240.41.196@tcp [8/256/0/180]
[ 7979.982954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7980.193258] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 7983.251942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7983.467350] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7984.987603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7984.992134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7985.221287] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7985.224136] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 7986.198111] Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod
[ 7986.428807] LNet: 170148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
[ 7986.430711] LNetError: 170148:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found
[ 7986.437842] LNet: Removed LNI 10.240.41.196@tcp
[ 7986.501800] Key type .llcrypt unregistered
[ 7986.502691] Key type ._llcrypt unregistered
[ 7989.570739] Key type ._llcrypt registered
[ 7989.574397] Key type .llcrypt registered
[ 7989.865188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7989.926555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7989.939992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7990.207499] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7990.242080] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7990.265298] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7990.334284] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
[ 7990.337640] alg: No test for adler32 (adler32-zlib)
[ 7992.337570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7992.496343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7992.566349] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7992.786031] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 7993.036114] Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure 
[ 7996.356848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 7996.568677] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 7999.737758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 7999.949279] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 8003.257672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 8003.472545] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 8006.788348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 8007.002678] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 8007.294886] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 8008.768243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 8008.776632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 8009.006155] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 8009.008860] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 8009.017079] LNet: Added LNI 10.240.41.196@tcp1 [8/256/0/180]
[ 8010.086026] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids
[ 8011.132214] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp --gateway 10.240.38.81@tcp1
[ 8011.310981] LNet: 172182:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled
[ 8011.489982] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp --gateway 10.240.41.156@tcp1
[ 8011.927728] Lustre: DEBUG MARKER: /usr/sbin/lctl show_route
[ 8012.288088] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route show -v
[ 8012.648670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n routes
[ 8013.014576] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping 10.240.39.188@tcp
[ 8013.373877] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 8014.912873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
[ 8014.934327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
[ 8014.952697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
[ 8015.209209] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
[ 8015.250816] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
[ 8015.262593] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
[ 8016.370662] LNet: 173151:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8016.471824] LNet: 173151:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8016.575814] LNet: 173151:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8016.679799] LNet: 173151:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8016.783793] LNet: 173151:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8016.887796] LNet: 173151:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8016.991809] LNet: 173151:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8017.654394] Lustre: DEBUG MARKER: Start LST rw
[ 8033.405641] Lustre: DEBUG MARKER: lst stop brw_rw
[ 8034.840801] Lustre: DEBUG MARKER: Stop LST rw
[ 8035.134048] Lustre: DEBUG MARKER: output="$(/usr/sbin/lnetctl route show --net tcp --gateway 10.240.38.81@tcp1 2>/dev/null)"; 		if [[ -n "${output}" ]]; then						echo "Delete route to tcp via 10.240.38.81@tcp1";				/usr/sbin/lnetctl route del --net tcp --gateway 10.240.38.81@tcp1;			else
[ 8035.528268] Lustre: DEBUG MARKER: output="$(/usr/sbin/lnetctl route show --net tcp --gateway 10.240.41.156@tcp1 2>/dev/null)"; 		if [[ -n "${output}" ]]; then						echo "Delete route to tcp via 10.240.41.156@tcp1";				/usr/sbin/lnetctl route del --net tcp --gateway 10.240.41.156@tcp1;			e
[ 8035.932262] Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod
[ 8036.192851] LNet: 173545:0:(timer.c:220:stt_shutdown()) waiting for 1 threads to terminate
[ 8036.294662] LNet: 173545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
[ 8036.294934] LNetError: 173545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found
[ 8037.348564] LNet: Removed LNI 10.240.41.196@tcp1
[ 8037.424858] Key type .llcrypt unregistered
[ 8037.425011] Key type ._llcrypt unregistered
[ 8202.856673] Autotest: Test running for 135 minutes (lustre-master_full-part-1_4681.175)
[ 8221.484247] Key type ._llcrypt registered
[ 8221.486289] Key type .llcrypt registered
[ 8221.755506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 8221.755506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 8221.988281] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 8221.988424] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet
[ 8222.099706] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
[ 8222.103239] alg: No test for adler32 (adler32-zlib)
[ 8224.197894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 8224.210559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 8224.477223] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet
[ 8224.494619] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet
[ 8224.825553] Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all
[ 8225.034498] LNet: Added LNI 10.240.41.196@tcp [8/256/0/180]
[ 8228.051378] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 8228.265563] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing lnet_if_list
[ 8231.396276] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 8231.609322] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing lnet_if_list
[ 8233.116327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 8233.116327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 8233.354603] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 8233.354603] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing lnet_if_list
[ 8234.325327] Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod
[ 8234.557705] LNet: 180320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
[ 8234.559712] LNetError: 180320:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found
[ 8234.566569] LNet: Removed LNI 10.240.41.196@tcp
[ 8234.630529] Key type .llcrypt unregistered
[ 8234.631421] Key type ._llcrypt unregistered
[ 8237.663602] Key type ._llcrypt registered
[ 8237.663788] Key type .llcrypt registered
[ 8237.962910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8237.989181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8238.062697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8238.301909] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8238.312056] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8238.341767] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8238.376649] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
[ 8238.379966] alg: No test for adler32 (adler32-zlib)
[ 8240.503278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8240.504348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8240.792805] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8240.804432] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9
[ 8241.128439] Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure 
[ 8244.417347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 8244.629075] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 8247.781981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 8247.999508] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 8251.342816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0
[ 8251.559258] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0
[ 8254.864242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 8255.078141] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 8255.372433] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 8256.841535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 8256.843991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0
[ 8257.067853] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 8257.080945] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0
[ 8257.088718] LNet: Added LNI 10.240.41.196@tcp1 [8/256/0/180]
[ 8258.162693] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids
[ 8259.240746] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp --gateway 10.240.38.81@tcp1
[ 8259.419360] LNet: 182355:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled
[ 8259.599754] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp --gateway 10.240.41.156@tcp1
[ 8260.040172] Lustre: DEBUG MARKER: /usr/sbin/lctl show_route
[ 8260.404819] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route show -v
[ 8260.770140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n routes
[ 8261.140322] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping 10.240.39.188@tcp
[ 8261.507924] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 8263.063784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
[ 8263.081528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
[ 8263.107394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
[ 8263.373275] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
[ 8263.399407] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
[ 8263.427902] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
[ 8264.486699] LNet: 183324:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8264.588539] LNet: 183324:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8264.692515] LNet: 183324:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8264.796516] LNet: 183324:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8264.900513] LNet: 183324:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8265.004503] LNet: 183324:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8265.108517] LNet: 183324:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
[ 8265.415794] Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval
[ 8265.828593] Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout
[ 8266.339560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start
[ 8266.590524] Lustre: DEBUG MARKER: Wait 5s for LST to start
[ 8266.657262] Lustre: DEBUG MARKER: Start LST rw
[ 8272.017000] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on trevis-24vm2
[ 8272.351095] Lustre: DEBUG MARKER: Disable routing on trevis-24vm2
[ 8273.062281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish
[ 8273.288564] Lustre: DEBUG MARKER: Wait for lst to finish
[ 8274.132134] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8274.132257] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) route to tcp through 10.240.38.81@tcp has gone from up to down
[ 8279.251966] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8282.362664] Lustre: DEBUG MARKER: lst stop brw_rw
[ 8284.371615] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8289.491289] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8294.611161] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8303.826704] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8303.826827] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message
[ 8324.305795] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8324.305874] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages
[ 8338.192746] LNet: 183325:0:(rpc.c:1099:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.240.39.188@tcp, timeout 64.
[ 8338.195354] LustreError: 45659:0:(brw_test.c:323:brw_client_done_rpc()) BRW RPC to 12345-10.240.39.188@tcp failed with -110
[ 8338.694389] Lustre: DEBUG MARKER: Stop LST rw
[ 8339.298081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity-lnet test_257: @@@@@@ FAIL: Detected 39 dropped messages - expect 0 
[ 8339.518036] Lustre: DEBUG MARKER: sanity-lnet test_257: @@@@@@ FAIL: Detected 39 dropped messages - expect 0
[ 8339.798968] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_4681_175_3e4a9bbf-3bb6-4eb4-addf-bb27972412b0//sanity-lnet.test_257.debug_log.$(hostname -s).1764886373.log;
		dmesg > /autotest/autotest-2/2025-12-04/lustre-master_full-part-1_
[ 8341.265305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:12:55 \(1764886375\)
[ 8341.487327] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:12:55 (1764886375)
[ 8359.120214] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8359.120296] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages
[ 8408.268101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:14:02 \(1764886442\)
[ 8408.490793] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:14:02 (1764886442)
[ 8410.564079] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) route to tcp through 10.240.41.156@tcp has gone from up to down
[ 8410.796030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:14:04 \(1764886444\)
[ 8411.018940] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:14:04 (1764886444)
[ 8412.838792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 270: Test NID metadata =============== 22:14:06 \(1764886446\)
[ 8413.063307] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:14:06 (1764886446)
[ 8417.445303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 280: Don\'t panic when request_module fails ========================================================== 22:14:11 \(1764886451\)
[ 8417.675630] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:14:11 (1764886451)
[ 8418.305629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_280 Need request_module to fail
[ 8418.529884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_280 Need request_module to fail
[ 8418.838770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:14:12 \(1764886452\)
[ 8419.059179] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:14:12 (1764886452)
[ 8424.653741] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8424.658348] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 12 previous similar messages
[ 8425.162011] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:14:19 \(1764886459\)
[ 8425.386683] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:14:19 (1764886459)
[ 8429.565015] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:14:23 \(1764886463\)
[ 8429.798200] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:14:23 (1764886463)
[ 8433.145865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 301: Check for dynamic adds of same\/wrong interface \(memory leak\) ========================================================== 22:14:27 \(1764886467\)
[ 8433.373206] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:14:27 (1764886467)
[ 8435.361077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:14:29 \(1764886469\)
[ 8435.589195] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:14:29 (1764886469)
[ 8438.036055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:14:32 \(1764886472\)
[ 8438.257807] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:14:32 (1764886472)
[ 8440.192190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_303 Need non-routed configuration
[ 8440.416068] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_303 Need non-routed configuration
[ 8440.730156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:14:34 \(1764886474\)
[ 8440.961656] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:14:34 (1764886474)
[ 8446.770477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:14:40 \(1764886480\)
[ 8446.993267] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:14:40 (1764886480)
[ 8449.421592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 310: Set timeout and verify ========== 22:14:43 \(1764886483\)
[ 8449.650558] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:14:43 (1764886483)
[ 8453.343922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:14:47 \(1764886487\)
[ 8453.569393] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:14:47 (1764886487)
[ 8453.841671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_311 Need kfi network type
[ 8454.061709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type
[ 8454.369711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:14:48 \(1764886488\)
[ 8454.594510] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:14:48 (1764886488)
[ 8454.868328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_312 Need kfi network type
[ 8455.090680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type
[ 8455.410280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 350: Check import --exec of ping\/discover ========================================================== 22:14:49 \(1764886489\)
[ 8455.633298] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:14:49 (1764886489)
[ 8457.300264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 400: Check for udsp add\/delete net rule without net num ========================================================== 22:14:51 \(1764886491\)
[ 8457.523895] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:14:51 (1764886491)
[ 8459.359458] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:14:53 \(1764886493\)
[ 8459.578996] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:14:53 (1764886493)
[ 8462.044305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:14:56 \(1764886496\)
[ 8462.266193] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:14:56 (1764886496)
[ 8464.789513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:14:58 \(1764886498\)
[ 8465.012165] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:14:58 (1764886498)
[ 8466.779738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 404: Check udsp src\/dst net rule ===== 22:15:00 \(1764886500\)
[ 8467.004676] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:15:00 (1764886500)
[ 8476.876370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410
[ 8477.120950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410
[ 8477.443751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:15:11 \(1764886511\)
[ 8477.668133] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:15:11 (1764886511)
[ 8479.581088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:15:13 \(1764886513\)
[ 8479.807124] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:15:13 (1764886513)
[ 8486.371731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 501: Verify lnetctl peer set --health \(SR\) ========================================================== 22:15:20 \(1764886520\)
[ 8486.594550] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:15:20 (1764886520)
[ 8489.389962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_501 Need non-routed configuration
[ 8489.619530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_501 Need non-routed configuration
[ 8489.933044] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 502: Verify lnetctl peer set --health \(MR\) ========================================================== 22:15:24 \(1764886524\)
[ 8490.155998] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:15:24 (1764886524)
[ 8492.129126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanity-lnet test_502 Need non-routed configuration
[ 8492.350560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_502 Need non-routed configuration
[ 8492.659297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:15:26 \(1764886526\)
[ 8492.885145] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:15:26 (1764886526)
[ 8494.429741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:15:28 \(1764886528\)
[ 8494.653294] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:15:28 (1764886528)
[ 8496.160893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:15:30 \(1764886530\)
[ 8496.385854] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:15:30 (1764886530)
[ 8497.873825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:15:31 \(1764886531\)
[ 8498.102071] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:15:31 (1764886531)
[ 8499.591760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:15:33 \(1764886533\)
[ 8499.822047] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:15:33 (1764886533)
[ 8501.334601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:15:35 \(1764886535\)
[ 8501.565378] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:15:35 (1764886535)
[ 8503.104722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:15:37 \(1764886537\)
[ 8503.326011] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:15:37 (1764886537)
[ 8503.494396] Autotest: Test running for 140 minutes (lustre-master_full-part-1_4681.175)
[ 8504.872653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:15:38 \(1764886538\)
[ 8505.094595] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:15:38 (1764886538)
[ 8506.633030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule \(missing IP pattern\) ========================================================== 22:15:40 \(1764886540\)
[ 8506.856990] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:15:40 (1764886540)
[ 8508.314319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:15:42 \(1764886542\)
[ 8508.540200] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:15:42 (1764886542)
[ 8510.003157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:15:44 \(1764886544\)
[ 8510.231582] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:15:44 (1764886544)
[ 8511.764791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:15:45 \(1764886545\)
[ 8511.987529] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:15:45 (1764886545)
[ 8513.426497] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:15:47 \(1764886547\)
[ 8513.651462] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:15:47 (1764886547)
[ 8515.099423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:15:49 \(1764886549\)
[ 8515.319088] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:15:49 (1764886549)
[ 8516.828955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:15:50 \(1764886550\)
[ 8517.056724] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:15:50 (1764886550)
[ 8518.585213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:15:52 \(1764886552\)
[ 8518.821894] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:15:52 (1764886552)
[ 8520.361701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:15:54 \(1764886554\)
[ 8520.584561] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:15:54 (1764886554)
[ 8522.082903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:15:56 \(1764886556\)
[ 8522.311532] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:15:56 (1764886556)
[ 8524.062225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:15:58 \(1764886558\)
[ 8524.285849] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:15:58 (1764886558)
[ 8525.773295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:15:59 \(1764886559\)
[ 8525.997453] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:15:59 (1764886559)
[ 8527.455490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:16:01 \(1764886561\)
[ 8527.682194] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:16:01 (1764886561)
[ 8529.173844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:16:03 \(1764886563\)
[ 8529.397738] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:16:03 (1764886563)
[ 8530.874288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:16:04 \(1764886564\)
[ 8531.100237] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:16:04 (1764886564)
[ 8532.656283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:16:06 \(1764886566\)
[ 8532.883973] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:16:06 (1764886566)
[ 8534.410778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:16:08 \(1764886568\)
[ 8534.637427] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:16:08 (1764886568)
[ 8536.171121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:16:10 \(1764886570\)
[ 8536.407039] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:16:10 (1764886570)
[ 8537.942985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:16:12 \(1764886572\)
[ 8538.165836] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:16:12 (1764886572)
[ 8539.694102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:16:13 \(1764886573\)
[ 8539.917406] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:16:13 (1764886573)
[ 8541.365099] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:16:15 \(1764886575\)
[ 8541.592108] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:16:15 (1764886575)
[ 8543.069592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:16:17 \(1764886577\)
[ 8543.299098] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:16:17 (1764886577)
[ 8544.786117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lnet test complete, duration 1591 sec ========== 22:16:18 \(1764886578\)
[ 8545.014385] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1591 sec ========== 22:16:18 (1764886578)
[ 8549.073556] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 8549.482618] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 8554.695801] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 10.240.38.81@tcp is being used as a gateway but routing feature is not turned on
[ 8554.695962] LNetError: 181288:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 25 previous similar messages
[ 8557.424697] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) route to tcp through 10.240.41.156@tcp has gone from down to up
[ 8560.285661] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8560.497722] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8567.229035] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) route to tcp through 10.240.41.156@tcp has gone from up to down
[ 8569.861302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8570.084343] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8577.992991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8578.214218] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8578.492403] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) route to tcp through 10.240.41.156@tcp has gone from up to down
[ 8578.492540] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) Skipped 1 previous similar message
[ 8586.171251] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8586.387988] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8589.758574] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) route to tcp through 10.240.41.156@tcp has gone from up to down
[ 8589.758654] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) Skipped 1 previous similar message
[ 8594.327502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8594.540562] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8602.469291] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8602.690015] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8610.540239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8610.755453] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8612.285221] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) route to tcp through 10.240.41.156@tcp has gone from up to down
[ 8612.285318] LNetError: 181288:0:(lib-lnet.h:1333:lnet_set_route_aliveness()) Skipped 3 previous similar messages
[ 8618.642514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8618.859038] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8620.977562] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
[ 8621.400871] Lustre: DEBUG MARKER: 
			running=$(mount | grep -c /mnt/lustre' ');
			rc=0;
			if [ $running -eq 0 ] ; then
				echo Mount client $(hostname): mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:/lustre /mnt/lustre
				mount -t lustre  -o user_xattr,flock 10.240.38.81@tcp:
[ 8621.638772] Lustre: Lustre: Build Version: 2.17.0_RC1
[ 8621.766612] LNetError: 195522:0:(peer.c:367:lnet_peer_ni_del_locked()) Peer NI 10.240.38.81@tcp is a gateway. Can not delete it
[ 8622.810985] Lustre: Mounted lustre-client
[ 8622.998535] Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
[ 8623.434980] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
[ 8624.922887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8624.922887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8625.149103] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8625.156533] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8627.199789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[ 8627.423278] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8627.700335] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[ 8631.600892] Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
[ 8631.954955] Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
[ 8637.433715] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: pjdfstest ============----- Thu Dec  4 10:17:51 PM UTC 2025
[ 8637.658940] Lustre: DEBUG MARKER: -----============= acceptance-small: pjdfstest ============----- Thu Dec 4 10:17:51 PM UTC 2025
[ 8645.538422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: chown_00 utimensat_08
[ 8645.760872] Lustre: DEBUG MARKER: excepting tests: chown_00 utimensat_08
[ 8646.048863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === run_pjdfstest: start setup 22:18:00 \(1764886680\) ===
[ 8646.264250] Lustre: DEBUG MARKER: === run_pjdfstest: start setup 22:18:00 (1764886680) ===
[ 8646.501660] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 8648.336477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-42vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 8648.702752] Lustre: DEBUG MARKER: trevis-42vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 8649.101439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 8649.161112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing check_config_client \/mnt\/lustre
[ 8649.474622] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1764886667/real 0]  req@ffff91fed0b0d040 x1850617790211712/t0(0) o400->lustre-OST0001-osc-ffff91ff8487a800@10.240.41.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1764886683 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8649.481420] Lustre: lustre-OST0005-osc-ffff91ff8487a800: Connection to lustre-OST0005 (at 10.240.41.156@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 8649.589223] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 8649.593226] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing check_config_client /mnt/lustre
[ 8653.210104] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 8653.614440] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts);
mpts=$(mount | grep -c /mnt/lustre2' ');
if [ $running -ne $mpts ]; then
	echo $(hostname) env are INSANE!;
	exit 1;
fi
[ 8654.594444] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1764886672/real 0]  req@ffff91fed3a7e3c0 x1850617790213248/t0(0) o400->lustre-OST0004-osc-ffff91ff8487a800@10.240.41.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1764886688 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8654.594601] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 8661.278539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
[ 8661.507033] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8661.774116] Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
[ 8664.358510] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
[ 8666.851694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8666.921568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8666.974241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8667.035942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8667.528976] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8667.595567] Lustre: DEBUG MARKER: trevis-66vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8667.687104] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8667.800405] Lustre: DEBUG MARKER: trevis-68vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8668.929715] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1764886685/real 0]  req@ffff91fed213d040 x1850617790218112/t0(0) o400->lustre-OST0003-osc-ffff91ff8487a800@10.240.41.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1764886703 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8669.245030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark === run_pjdfstest: finish setup 22:18:23 \(1764886703\) ===
[ 8669.476042] Lustre: DEBUG MARKER: === run_pjdfstest: finish setup 22:18:23 (1764886703) ===
[ 8670.977623] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1764886687/real 0]  req@ffff91fed3a7ea40 x1850617790220160/t0(0) o400->lustre-OST0003-osc-ffff91ff8487a800@10.240.41.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1764886705 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8672.345406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing add_group 65535 pjd_grp1
[ 8672.557207] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing add_group 65535 pjd_grp1
[ 8674.918881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing add_group 65533 pjd_grp2
[ 8675.139102] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing add_group 65533 pjd_grp2
[ 8677.121533] Lustre: 195531:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1764886693/real 0]  req@ffff91fed213c340 x1850617790225152/t0(0) o400->lustre-OST0001-osc-ffff91ff8487a800@10.240.41.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1764886711 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8677.466141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing add_group 65531 pjd_grp3
[ 8677.676744] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing add_group 65531 pjd_grp3
[ 8679.947375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing add_user 65535 pjd_usr1 65535 \/mnt\/lustre\/pjd_usr1
[ 8680.159843] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing add_user 65535 pjd_usr1 65535 /mnt/lustre/pjd_usr1
[ 8682.560640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing add_user 65533 pjd_usr2 65533 \/mnt\/lustre\/pjd_usr2
[ 8682.770236] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing add_user 65533 pjd_usr2 65533 /mnt/lustre/pjd_usr2
[ 8685.154799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm2.trevis.whamcloud.com: executing add_user 65532 pjd_usr3 65531 \/mnt\/lustre\/pjd_usr3
[ 8685.367588] Lustre: DEBUG MARKER: trevis-24vm2.trevis.whamcloud.com: executing add_user 65532 pjd_usr3 65531 /mnt/lustre/pjd_usr3
[ 8685.854477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_00: chflags changes flags == 22:18:39 \(1764886719\)
[ 8686.077353] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_00: chflags changes flags == 22:18:39 (1764886719)
[ 8686.356283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/00.t against ext4 filesystem
[ 8686.576035] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/00.t against ext4 filesystem
[ 8687.226275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/00.t against lustre filesystem
[ 8687.447163] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/00.t against lustre filesystem
[ 8687.893832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8688.116057] Lustre: DEBUG MARKER: ext4 report
[ 8688.384843] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1764886702/real 0]  req@ffff91fed2217a80 x1850617790235904/t0(0) o400->lustre-OST0000-osc-ffff91ff8487a800@10.240.41.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1764886722 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8688.384955] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 8688.400934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8688.627612] Lustre: DEBUG MARKER: lustre report
[ 8690.033272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_01: chflags returns ENOTDIR if a component of the path prefix is not a directory ========================================================== 22:18:44 \(1764886724\)
[ 8690.262282] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_01: chflags returns ENOTDIR if a component of the path prefix is not a directory ========================================================== 22:18:44 (1764886724)
[ 8690.539794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/01.t against ext4 filesystem
[ 8690.761777] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/01.t against ext4 filesystem
[ 8691.194931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/01.t against lustre filesystem
[ 8691.412553] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/01.t against lustre filesystem
[ 8691.857519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8692.080081] Lustre: DEBUG MARKER: ext4 report
[ 8692.351848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8692.573130] Lustre: DEBUG MARKER: lustre report
[ 8694.002910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_02: chflags returns ENAMETOOLONG if a component of a pathname exceeded {NAME_MAX} characters ========================================================== 22:18:48 \(1764886728\)
[ 8694.225232] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_02: chflags returns ENAMETOOLONG if a component of a pathname exceeded {NAME_MAX} characters ========================================================== 22:18:48 (1764886728)
[ 8694.496477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/02.t against ext4 filesystem
[ 8694.724677] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/02.t against ext4 filesystem
[ 8695.156756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/02.t against lustre filesystem
[ 8695.378976] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/02.t against lustre filesystem
[ 8695.818071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8696.038949] Lustre: DEBUG MARKER: ext4 report
[ 8696.314112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8696.533323] Lustre: DEBUG MARKER: lustre report
[ 8697.913338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_03: chflags returns ENAMETOOLONG if an entire path name exceeded {PATH_MAX} characters ========================================================== 22:18:52 \(1764886732\)
[ 8698.136031] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_03: chflags returns ENAMETOOLONG if an entire path name exceeded {PATH_MAX} characters ========================================================== 22:18:52 (1764886732)
[ 8698.411707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/03.t against ext4 filesystem
[ 8698.631411] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/03.t against ext4 filesystem
[ 8699.061164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/03.t against lustre filesystem
[ 8699.288617] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/03.t against lustre filesystem
[ 8699.732449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8699.954607] Lustre: DEBUG MARKER: ext4 report
[ 8700.225626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8700.452856] Lustre: DEBUG MARKER: lustre report
[ 8701.884712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_04: chflags returns ENOENT if the named file does not exist ========================================================== 22:18:55 \(1764886735\)
[ 8702.107461] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_04: chflags returns ENOENT if the named file does not exist ========================================================== 22:18:55 (1764886735)
[ 8702.383788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/04.t against ext4 filesystem
[ 8702.608758] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/04.t against ext4 filesystem
[ 8703.034551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/04.t against lustre filesystem
[ 8703.253297] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/04.t against lustre filesystem
[ 8703.696133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8703.916291] Lustre: DEBUG MARKER: ext4 report
[ 8704.191318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8704.411704] Lustre: DEBUG MARKER: lustre report
[ 8705.796653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_05: chflags returns EACCES when search permission is denied for a component of the path prefix ========================================================== 22:18:59 \(1764886739\)
[ 8706.020820] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_05: chflags returns EACCES when search permission is denied for a component of the path prefix ========================================================== 22:18:59 (1764886739)
[ 8706.298830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/05.t against ext4 filesystem
[ 8706.528235] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/05.t against ext4 filesystem
[ 8706.955473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/05.t against lustre filesystem
[ 8707.178567] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/05.t against lustre filesystem
[ 8707.613742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8707.838815] Lustre: DEBUG MARKER: ext4 report
[ 8708.114150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8708.340088] Lustre: DEBUG MARKER: lustre report
[ 8709.729491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_06: chflags returns ELOOP if too many symbolic links were encountered in translating the pathname ========================================================== 22:19:03 \(1764886743\)
[ 8709.951462] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_06: chflags returns ELOOP if too many symbolic links were encountered in translating the pathname ========================================================== 22:19:03 (1764886743)
[ 8710.232106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/06.t against ext4 filesystem
[ 8710.454477] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/06.t against ext4 filesystem
[ 8710.879444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/06.t against lustre filesystem
[ 8711.099981] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/06.t against lustre filesystem
[ 8711.576067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8711.792932] Lustre: DEBUG MARKER: ext4 report
[ 8712.071151] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8712.293479] Lustre: DEBUG MARKER: lustre report
[ 8713.685334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_07: chflags returns EPERM when the effective user ID does not match the owner of the file and the effective user ID is not the super-user ========================================================== 22:19:07
[ 8713.913137] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_07: chflags returns EPERM when the effective user ID does not match the owner of the file and the effective user ID is not the super-user ========================================================== 22:19:07 (1764886747)
[ 8714.192148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/07.t against ext4 filesystem
[ 8714.426892] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/07.t against ext4 filesystem
[ 8714.850976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/07.t against lustre filesystem
[ 8715.073833] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/07.t against lustre filesystem
[ 8715.502493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8715.727356] Lustre: DEBUG MARKER: ext4 report
[ 8715.999696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8716.221440] Lustre: DEBUG MARKER: lustre report
[ 8717.617188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_08: chflags returns EPERM when one of SF_IMMUTABLE, SF_APPEND, or SF_NOUNLINK is set and the user is not the super-user ========================================================== 22:19:11 \(1764886751\)
[ 8717.846394] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_08: chflags returns EPERM when one of SF_IMMUTABLE, SF_APPEND, or SF_NOUNLINK is set and the user is not the super-user ========================================================== 22:19:11 (1764886751)
[ 8718.118903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/08.t against ext4 filesystem
[ 8718.342325] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/08.t against ext4 filesystem
[ 8718.765895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/08.t against lustre filesystem
[ 8718.987257] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/08.t against lustre filesystem
[ 8719.417959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8719.645313] Lustre: DEBUG MARKER: ext4 report
[ 8719.919052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8720.139653] Lustre: DEBUG MARKER: lustre report
[ 8721.523044] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_09: chflags returns EPERM when one of SF_IMMUTABLE, SF_APPEND, or SF_NOUNLINK is set and securelevel is greater than 0 ========================================================== 22:19:15 \(1764886755\)
[ 8721.749143] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_09: chflags returns EPERM when one of SF_IMMUTABLE, SF_APPEND, or SF_NOUNLINK is set and securelevel is greater than 0 ========================================================== 22:19:15 (1764886755)
[ 8722.027647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/09.t against ext4 filesystem
[ 8722.175306] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1764886727/real 0]  req@ffff91fed2216700 x1850617790259456/t0(0) o400->lustre-OST0003-osc-ffff91ff8487a800@10.240.41.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1764886756 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8722.175402] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 8722.255663] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/09.t against ext4 filesystem
[ 8722.682530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/09.t against lustre filesystem
[ 8722.902956] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/09.t against lustre filesystem
[ 8723.338805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8723.556533] Lustre: DEBUG MARKER: ext4 report
[ 8723.828932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8724.050106] Lustre: DEBUG MARKER: lustre report
[ 8725.406473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_10: chflags returns EPERM if non-super-user tries to set one of SF_IMMUTABLE, SF_APPEND, or SF_NOUNLINK ========================================================== 22:19:19 \(1764886759\)
[ 8725.625254] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_10: chflags returns EPERM if non-super-user tries to set one of SF_IMMUTABLE, SF_APPEND, or SF_NOUNLINK ========================================================== 22:19:19 (1764886759)
[ 8725.908344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/10.t against ext4 filesystem
[ 8726.125206] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/10.t against ext4 filesystem
[ 8726.555501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/10.t against lustre filesystem
[ 8726.771873] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/10.t against lustre filesystem
[ 8727.209934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8727.426066] Lustre: DEBUG MARKER: ext4 report
[ 8727.702676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8727.922348] Lustre: DEBUG MARKER: lustre report
[ 8729.325491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_11: chflags returns EPERM if a user tries to set or remove the SF_SNAPSHOT flag ========================================================== 22:19:23 \(1764886763\)
[ 8729.551057] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_11: chflags returns EPERM if a user tries to set or remove the SF_SNAPSHOT flag ========================================================== 22:19:23 (1764886763)
[ 8729.828383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/11.t against ext4 filesystem
[ 8730.047969] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/11.t against ext4 filesystem
[ 8730.472606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/11.t against lustre filesystem
[ 8730.692887] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/11.t against lustre filesystem
[ 8731.132452] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8731.346105] Lustre: DEBUG MARKER: ext4 report
[ 8731.620496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8731.839355] Lustre: DEBUG MARKER: lustre report
[ 8733.221626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_12: chflags returns EROFS if the named file resides on a read-only file system ========================================================== 22:19:27 \(1764886767\)
[ 8733.448423] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_12: chflags returns EROFS if the named file resides on a read-only file system ========================================================== 22:19:27 (1764886767)
[ 8733.724198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/12.t against ext4 filesystem
[ 8733.947377] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/12.t against ext4 filesystem
[ 8734.371906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/12.t against lustre filesystem
[ 8734.592871] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/12.t against lustre filesystem
[ 8735.029539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8735.249032] Lustre: DEBUG MARKER: ext4 report
[ 8735.523382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8735.743156] Lustre: DEBUG MARKER: lustre report
[ 8737.121477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chflags_13: chflags returns EFAULT if the path argument points outside the process\'s allocated address space ========================================================== 22:19:31 \(1764886771\)
[ 8737.343561] Lustre: DEBUG MARKER: == run_pjdfstest test chflags_13: chflags returns EFAULT if the path argument points outside the process's allocated address space ========================================================== 22:19:31 (1764886771)
[ 8737.615864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/13.t against ext4 filesystem
[ 8737.837161] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/13.t against ext4 filesystem
[ 8738.258159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chflags\/13.t against lustre filesystem
[ 8738.479618] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chflags/13.t against lustre filesystem
[ 8738.916066] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8739.136321] Lustre: DEBUG MARKER: ext4 report
[ 8739.405405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8739.630393] Lustre: DEBUG MARKER: lustre report
[ 8741.037410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_00: chmod changes permission ========================================================== 22:19:35 \(1764886775\)
[ 8741.261262] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_00: chmod changes permission ========================================================== 22:19:35 (1764886775)
[ 8741.551892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/00.t against ext4 filesystem
[ 8741.769181] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/00.t against ext4 filesystem
[ 8754.855900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/00.t against lustre filesystem
[ 8755.079913] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/00.t against lustre filesystem
[ 8768.280240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8768.500133] Lustre: DEBUG MARKER: ext4 report
[ 8768.776502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8768.998403] Lustre: DEBUG MARKER: lustre report
[ 8770.393732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_01: chmod returns ENOTDIR if a component of the path prefix is not a directory ========================================================== 22:20:04 \(1764886804\)
[ 8770.615714] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_01: chmod returns ENOTDIR if a component of the path prefix is not a directory ========================================================== 22:20:04 (1764886804)
[ 8770.898669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/01.t against ext4 filesystem
[ 8771.118991] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/01.t against ext4 filesystem
[ 8771.647604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/01.t against lustre filesystem
[ 8771.877517] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/01.t against lustre filesystem
[ 8772.434231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8772.656319] Lustre: DEBUG MARKER: ext4 report
[ 8772.933967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8773.160180] Lustre: DEBUG MARKER: lustre report
[ 8774.544461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_02: chmod returns ENAMETOOLONG if a component of a pathname exceeded {NAME_MAX} characters ========================================================== 22:20:08 \(1764886808\)
[ 8774.771349] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_02: chmod returns ENAMETOOLONG if a component of a pathname exceeded {NAME_MAX} characters ========================================================== 22:20:08 (1764886808)
[ 8775.042127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/02.t against ext4 filesystem
[ 8775.263802] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/02.t against ext4 filesystem
[ 8775.793954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/02.t against lustre filesystem
[ 8776.015925] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/02.t against lustre filesystem
[ 8776.568059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8776.782294] Lustre: DEBUG MARKER: ext4 report
[ 8777.062749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8777.283296] Lustre: DEBUG MARKER: lustre report
[ 8778.715135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_03: chmod returns ENAMETOOLONG if an entire path name exceeded {PATH_MAX} characters ========================================================== 22:20:12 \(1764886812\)
[ 8778.939426] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_03: chmod returns ENAMETOOLONG if an entire path name exceeded {PATH_MAX} characters ========================================================== 22:20:12 (1764886812)
[ 8779.208425] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/03.t against ext4 filesystem
[ 8779.425165] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/03.t against ext4 filesystem
[ 8781.283863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/03.t against lustre filesystem
[ 8781.506259] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/03.t against lustre filesystem
[ 8783.551971] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8783.774645] Lustre: DEBUG MARKER: ext4 report
[ 8784.047803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8784.266117] Lustre: DEBUG MARKER: lustre report
[ 8785.676745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_04: chmod returns ENOENT if the named file does not exist ========================================================== 22:20:19 \(1764886819\)
[ 8785.897993] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_04: chmod returns ENOENT if the named file does not exist ========================================================== 22:20:19 (1764886819)
[ 8786.169943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/04.t against ext4 filesystem
[ 8786.391501] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/04.t against ext4 filesystem
[ 8786.874041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/04.t against lustre filesystem
[ 8787.092906] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/04.t against lustre filesystem
[ 8787.591525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8787.812691] Lustre: DEBUG MARKER: ext4 report
[ 8788.086327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8788.308063] Lustre: DEBUG MARKER: lustre report
[ 8789.724162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_05: chmod returns EACCES when search permission is denied for a component of the path prefix ========================================================== 22:20:23 \(1764886823\)
[ 8789.950059] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_05: chmod returns EACCES when search permission is denied for a component of the path prefix ========================================================== 22:20:23 (1764886823)
[ 8790.225622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/05.t against ext4 filesystem
[ 8790.441973] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/05.t against ext4 filesystem
[ 8790.963945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/05.t against lustre filesystem
[ 8791.186920] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/05.t against lustre filesystem
[ 8791.742173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8791.964332] Lustre: DEBUG MARKER: ext4 report
[ 8792.241447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8792.461571] Lustre: DEBUG MARKER: lustre report
[ 8793.888625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_06: chmod returns ELOOP if too many symbolic links were encountered in translating the pathname ========================================================== 22:20:28 \(1764886828\)
[ 8794.110877] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_06: chmod returns ELOOP if too many symbolic links were encountered in translating the pathname ========================================================== 22:20:28 (1764886828)
[ 8794.387778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/06.t against ext4 filesystem
[ 8794.605111] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/06.t against ext4 filesystem
[ 8795.095102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/06.t against lustre filesystem
[ 8795.315113] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/06.t against lustre filesystem
[ 8795.818868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8796.039306] Lustre: DEBUG MARKER: ext4 report
[ 8796.313399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8796.535600] Lustre: DEBUG MARKER: lustre report
[ 8797.912787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_07: chmod returns EPERM if the operation would change the ownership, but the effective user ID is not the super-user ========================================================== 22:20:32 \(1764886832\)
[ 8798.135494] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_07: chmod returns EPERM if the operation would change the ownership, but the effective user ID is not the super-user ========================================================== 22:20:32 (1764886832)
[ 8798.413072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/07.t against ext4 filesystem
[ 8798.633243] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/07.t against ext4 filesystem
[ 8799.211329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/07.t against lustre filesystem
[ 8799.431780] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/07.t against lustre filesystem
[ 8800.053279] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8800.273962] Lustre: DEBUG MARKER: ext4 report
[ 8800.548665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8800.767236] Lustre: DEBUG MARKER: lustre report
[ 8802.135996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_08: chmod returns EPERM if the named file has its immutable or append-only flag set ========================================================== 22:20:36 \(1764886836\)
[ 8802.360439] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_08: chmod returns EPERM if the named file has its immutable or append-only flag set ========================================================== 22:20:36 (1764886836)
[ 8802.634537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/08.t against ext4 filesystem
[ 8802.858316] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/08.t against ext4 filesystem
[ 8803.285557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/08.t against lustre filesystem
[ 8803.508934] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/08.t against lustre filesystem
[ 8803.952281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8804.171583] Lustre: DEBUG MARKER: ext4 report
[ 8804.446171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8804.836448] Autotest: Test running for 145 minutes (lustre-master_full-part-1_4681.175)
[ 8804.846786] Lustre: DEBUG MARKER: lustre report
[ 8805.115537] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1764886806/real 0]  req@ffff91fed213f0c0 x1850617790319744/t0(0) o400->lustre-OST0000-osc-ffff91ff8487a800@10.240.41.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1764886839 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8805.115643] Lustre: 195530:0:(client.c:2472:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 8806.250279] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_09: chmod returns EROFS if the named file resides on a read-only file system ========================================================== 22:20:40 \(1764886840\)
[ 8806.472376] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_09: chmod returns EROFS if the named file resides on a read-only file system ========================================================== 22:20:40 (1764886840)
[ 8806.749054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/09.t against ext4 filesystem
[ 8806.970526] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/09.t against ext4 filesystem
[ 8807.398375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/09.t against lustre filesystem
[ 8807.619599] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/09.t against lustre filesystem
[ 8808.055895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8808.280173] Lustre: DEBUG MARKER: ext4 report
[ 8808.552899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8808.771837] Lustre: DEBUG MARKER: lustre report
[ 8810.147911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_10: chmod returns EFAULT if the path argument points outside the process\'s allocated address space ========================================================== 22:20:44 \(1764886844\)
[ 8810.371193] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_10: chmod returns EFAULT if the path argument points outside the process's allocated address space ========================================================== 22:20:44 (1764886844)
[ 8810.646892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/10.t against ext4 filesystem
[ 8810.882770] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/10.t against ext4 filesystem
[ 8811.324476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/10.t against lustre filesystem
[ 8811.545259] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/10.t against lustre filesystem
[ 8811.992886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8812.210883] Lustre: DEBUG MARKER: ext4 report
[ 8812.487437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8812.707391] Lustre: DEBUG MARKER: lustre report
[ 8814.097456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_11: chmod returns EFTYPE if the effective user ID is not the super-user, the mode includes the sticky bit \(S_ISVTX\), and path does not refer to a directory =================================================
[ 8814.320757] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_11: chmod returns EFTYPE if the effective user ID is not the super-user, the mode includes the sticky bit (S_ISVTX), and path does not refer to a directory ========================================================== 22:20:48 (17
[ 8814.598705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/11.t against ext4 filesystem
[ 8814.820614] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/11.t against ext4 filesystem
[ 8815.836213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/11.t against lustre filesystem
[ 8816.057814] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/11.t against lustre filesystem
[ 8817.162080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8817.377066] Lustre: DEBUG MARKER: ext4 report
[ 8817.658388] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8817.879660] Lustre: DEBUG MARKER: lustre report
[ 8819.321809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chmod_12: verify SUID\/SGID bit behaviour ========================================================== 22:20:53 \(1764886853\)
[ 8819.541727] Lustre: DEBUG MARKER: == run_pjdfstest test chmod_12: verify SUID/SGID bit behaviour ========================================================== 22:20:53 (1764886853)
[ 8819.818958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/12.t against ext4 filesystem
[ 8820.045936] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/12.t against ext4 filesystem
[ 8820.597280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chmod\/12.t against lustre filesystem
[ 8820.819297] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chmod/12.t against lustre filesystem
[ 8821.390986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8821.605045] Lustre: DEBUG MARKER: ext4 report
[ 8821.891339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8822.113866] Lustre: DEBUG MARKER: lustre report
[ 8823.508798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: run_pjdfstest test_chown_00 skipping ALWAYS excluded test chown_00
[ 8823.732926] Lustre: DEBUG MARKER: SKIP: run_pjdfstest test_chown_00 skipping ALWAYS excluded test chown_00
[ 8824.025339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_01: chown returns ENOTDIR if a component of the path prefix is not a directory ========================================================== 22:20:58 \(1764886858\)
[ 8824.254089] Lustre: DEBUG MARKER: == run_pjdfstest test chown_01: chown returns ENOTDIR if a component of the path prefix is not a directory ========================================================== 22:20:58 (1764886858)
[ 8824.526762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/01.t against ext4 filesystem
[ 8824.752636] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/01.t against ext4 filesystem
[ 8825.310819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/01.t against lustre filesystem
[ 8825.534887] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/01.t against lustre filesystem
[ 8826.120785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8826.345915] Lustre: DEBUG MARKER: ext4 report
[ 8826.625059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8826.847114] Lustre: DEBUG MARKER: lustre report
[ 8828.289603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_02: chown returns ENAMETOOLONG if a component of a pathname exceeded  characters ========================================================== 22:21:02 \(1764886862\)
[ 8828.519944] Lustre: DEBUG MARKER: == run_pjdfstest test chown_02: chown returns ENAMETOOLONG if a component of a pathname exceeded characters ========================================================== 22:21:02 (1764886862)
[ 8828.800102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/02.t against ext4 filesystem
[ 8829.020664] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/02.t against ext4 filesystem
[ 8829.583640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/02.t against lustre filesystem
[ 8829.809717] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/02.t against lustre filesystem
[ 8830.400234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8830.624189] Lustre: DEBUG MARKER: ext4 report
[ 8830.898552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8831.123990] Lustre: DEBUG MARKER: lustre report
[ 8832.553749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_03: chown returns ENAMETOOLONG if an entire path name exceeded {PATH_MAX} characters ========================================================== 22:21:06 \(1764886866\)
[ 8832.800450] Lustre: DEBUG MARKER: == run_pjdfstest test chown_03: chown returns ENAMETOOLONG if an entire path name exceeded {PATH_MAX} characters ========================================================== 22:21:06 (1764886866)
[ 8833.079173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/03.t against ext4 filesystem
[ 8833.306379] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/03.t against ext4 filesystem
[ 8835.193907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/03.t against lustre filesystem
[ 8835.417620] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/03.t against lustre filesystem
[ 8837.524572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8837.742936] Lustre: DEBUG MARKER: ext4 report
[ 8838.020688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8838.242309] Lustre: DEBUG MARKER: lustre report
[ 8839.653241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_04: chown returns ENOENT if the named file does not exist ========================================================== 22:21:13 \(1764886873\)
[ 8839.882325] Lustre: DEBUG MARKER: == run_pjdfstest test chown_04: chown returns ENOENT if the named file does not exist ========================================================== 22:21:13 (1764886873)
[ 8840.164140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/04.t against ext4 filesystem
[ 8840.385520] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/04.t against ext4 filesystem
[ 8840.879791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/04.t against lustre filesystem
[ 8841.104688] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/04.t against lustre filesystem
[ 8841.622626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8841.840117] Lustre: DEBUG MARKER: ext4 report
[ 8842.113451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8842.334156] Lustre: DEBUG MARKER: lustre report
[ 8843.749746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_05: chown returns EACCES when search permission is denied for a component of the path prefix ========================================================== 22:21:17 \(1764886877\)
[ 8843.975847] Lustre: DEBUG MARKER: == run_pjdfstest test chown_05: chown returns EACCES when search permission is denied for a component of the path prefix ========================================================== 22:21:17 (1764886877)
[ 8844.252662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/05.t against ext4 filesystem
[ 8844.473524] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/05.t against ext4 filesystem
[ 8845.012988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/05.t against lustre filesystem
[ 8845.238624] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/05.t against lustre filesystem
[ 8845.808533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8846.026775] Lustre: DEBUG MARKER: ext4 report
[ 8846.305371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8846.525293] Lustre: DEBUG MARKER: lustre report
[ 8847.966987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_06: chown returns ELOOP if too many symbolic links were encountered in translating the pathname ========================================================== 22:21:22 \(1764886882\)
[ 8848.193954] Lustre: DEBUG MARKER: == run_pjdfstest test chown_06: chown returns ELOOP if too many symbolic links were encountered in translating the pathname ========================================================== 22:21:22 (1764886882)
[ 8848.469636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/06.t against ext4 filesystem
[ 8848.693050] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/06.t against ext4 filesystem
[ 8849.187300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/06.t against lustre filesystem
[ 8849.411046] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/06.t against lustre filesystem
[ 8849.930559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8850.148802] Lustre: DEBUG MARKER: ext4 report
[ 8850.427906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8850.647498] Lustre: DEBUG MARKER: lustre report
[ 8852.070774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_07: chown returns EPERM if the operation would change the ownership, but the effective user ID is not the super-user and the process is not an owner of the file ==============================================
[ 8852.296163] Lustre: DEBUG MARKER: == run_pjdfstest test chown_07: chown returns EPERM if the operation would change the ownership, but the effective user ID is not the super-user and the process is not an owner of the file ========================================================== 22:21:2
[ 8852.575555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/07.t against ext4 filesystem
[ 8852.805510] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/07.t against ext4 filesystem
[ 8853.944110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/07.t against lustre filesystem
[ 8854.166768] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/07.t against lustre filesystem
[ 8855.372763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8855.596141] Lustre: DEBUG MARKER: ext4 report
[ 8855.874224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8856.100454] Lustre: DEBUG MARKER: lustre report
[ 8857.533653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_08: chown returns EPERM if the named file has its immutable or append-only flag set ========================================================== 22:21:31 \(1764886891\)
[ 8857.757787] Lustre: DEBUG MARKER: == run_pjdfstest test chown_08: chown returns EPERM if the named file has its immutable or append-only flag set ========================================================== 22:21:31 (1764886891)
[ 8858.030862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/08.t against ext4 filesystem
[ 8858.255455] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/08.t against ext4 filesystem
[ 8858.691904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/08.t against lustre filesystem
[ 8858.917512] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/08.t against lustre filesystem
[ 8859.360688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8859.588943] Lustre: DEBUG MARKER: ext4 report
[ 8859.866963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8860.086404] Lustre: DEBUG MARKER: lustre report
[ 8861.497843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_09: chown returns EROFS if the named file resides on a read-only file system ========================================================== 22:21:35 \(1764886895\)
[ 8861.720617] Lustre: DEBUG MARKER: == run_pjdfstest test chown_09: chown returns EROFS if the named file resides on a read-only file system ========================================================== 22:21:35 (1764886895)
[ 8861.990746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/09.t against ext4 filesystem
[ 8862.221090] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/09.t against ext4 filesystem
[ 8862.665243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/09.t against lustre filesystem
[ 8862.886842] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/09.t against lustre filesystem
[ 8863.326004] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8863.546213] Lustre: DEBUG MARKER: ext4 report
[ 8863.819425] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8864.040262] Lustre: DEBUG MARKER: lustre report
[ 8865.463709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test chown_10: chown returns EFAULT if the path argument points outside the process\'s allocated address space ========================================================== 22:21:39 \(1764886899\)
[ 8865.691652] Lustre: DEBUG MARKER: == run_pjdfstest test chown_10: chown returns EFAULT if the path argument points outside the process's allocated address space ========================================================== 22:21:39 (1764886899)
[ 8865.968964] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/10.t against ext4 filesystem
[ 8866.198503] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/10.t against ext4 filesystem
[ 8866.644119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/chown\/10.t against lustre filesystem
[ 8866.862313] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/chown/10.t against lustre filesystem
[ 8867.327532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8867.547825] Lustre: DEBUG MARKER: ext4 report
[ 8867.823592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8868.041181] Lustre: DEBUG MARKER: lustre report
[ 8869.440970] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_00: ftruncate descrease\/increase file size ========================================================== 22:21:43 \(1764886903\)
[ 8869.663862] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_00: ftruncate descrease/increase file size ========================================================== 22:21:43 (1764886903)
[ 8869.935661] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/00.t against ext4 filesystem
[ 8870.157434] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/00.t against ext4 filesystem
[ 8872.728733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/00.t against lustre filesystem
[ 8872.952888] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/00.t against lustre filesystem
[ 8876.016131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8876.241514] Lustre: DEBUG MARKER: ext4 report
[ 8876.513387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8876.734519] Lustre: DEBUG MARKER: lustre report
[ 8878.150379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_01: truncate returns ENOTDIR if a component of the path prefix is not a directory ========================================================== 22:21:52 \(1764886912\)
[ 8878.370993] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_01: truncate returns ENOTDIR if a component of the path prefix is not a directory ========================================================== 22:21:52 (1764886912)
[ 8878.648592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/01.t against ext4 filesystem
[ 8878.866678] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/01.t against ext4 filesystem
[ 8879.338504] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/01.t against lustre filesystem
[ 8879.559605] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/01.t against lustre filesystem
[ 8880.046467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8880.270052] Lustre: DEBUG MARKER: ext4 report
[ 8880.544870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8880.763750] Lustre: DEBUG MARKER: lustre report
[ 8882.149120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_02: truncate returns ENAMETOOLONG if a component of a pathname exceeded {NAME_MAX} characters ========================================================== 22:21:56 \(1764886916\)
[ 8882.378406] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_02: truncate returns ENAMETOOLONG if a component of a pathname exceeded {NAME_MAX} characters ========================================================== 22:21:56 (1764886916)
[ 8882.655028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/02.t against ext4 filesystem
[ 8882.874323] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/02.t against ext4 filesystem
[ 8883.404911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/02.t against lustre filesystem
[ 8883.625730] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/02.t against lustre filesystem
[ 8884.178384] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8884.398821] Lustre: DEBUG MARKER: ext4 report
[ 8884.672381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8884.893290] Lustre: DEBUG MARKER: lustre report
[ 8886.304346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_03: truncate returns ENAMETOOLONG if an entire path name exceeded {PATH_MAX} characters ========================================================== 22:22:00 \(1764886920\)
[ 8886.524568] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_03: truncate returns ENAMETOOLONG if an entire path name exceeded {PATH_MAX} characters ========================================================== 22:22:00 (1764886920)
[ 8886.807204] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/03.t against ext4 filesystem
[ 8887.031683] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/03.t against ext4 filesystem
[ 8888.887522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/03.t against lustre filesystem
[ 8889.108896] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/03.t against lustre filesystem
[ 8891.162507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8891.381665] Lustre: DEBUG MARKER: ext4 report
[ 8891.664007] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8891.886715] Lustre: DEBUG MARKER: lustre report
[ 8893.307673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_04: truncate returns ENOENT if the named file does not exist ========================================================== 22:22:07 \(1764886927\)
[ 8893.533037] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_04: truncate returns ENOENT if the named file does not exist ========================================================== 22:22:07 (1764886927)
[ 8893.804125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/04.t against ext4 filesystem
[ 8894.023894] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/04.t against ext4 filesystem
[ 8894.484253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/04.t against lustre filesystem
[ 8894.701960] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/04.t against lustre filesystem
[ 8895.184392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8895.406550] Lustre: DEBUG MARKER: ext4 report
[ 8895.679152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8895.903869] Lustre: DEBUG MARKER: lustre report
[ 8897.309738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_05: truncate returns EACCES when search permission is denied for a component of the path prefix ========================================================== 22:22:11 \(1764886931\)
[ 8897.535128] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_05: truncate returns EACCES when search permission is denied for a component of the path prefix ========================================================== 22:22:11 (1764886931)
[ 8897.812382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/05.t against ext4 filesystem
[ 8898.031830] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/05.t against ext4 filesystem
[ 8898.554555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/05.t against lustre filesystem
[ 8898.777506] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/05.t against lustre filesystem
[ 8899.334765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8899.552940] Lustre: DEBUG MARKER: ext4 report
[ 8899.831072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8900.055648] Lustre: DEBUG MARKER: lustre report
[ 8901.498457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_06: truncate returns EACCES if the named file is not writable by the user ========================================================== 22:22:15 \(1764886935\)
[ 8901.727089] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_06: truncate returns EACCES if the named file is not writable by the user ========================================================== 22:22:15 (1764886935)
[ 8902.015903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/06.t against ext4 filesystem
[ 8902.242552] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/06.t against ext4 filesystem
[ 8902.730118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/06.t against lustre filesystem
[ 8902.956529] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/06.t against lustre filesystem
[ 8903.474282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8903.693617] Lustre: DEBUG MARKER: ext4 report
[ 8903.969549] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8904.196407] Lustre: DEBUG MARKER: lustre report
[ 8905.647686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_07: truncate returns ELOOP if too many symbolic links were encountered in translating the pathname ========================================================== 22:22:19 \(1764886939\)
[ 8905.876577] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_07: truncate returns ELOOP if too many symbolic links were encountered in translating the pathname ========================================================== 22:22:19 (1764886939)
[ 8906.161364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/07.t against ext4 filesystem
[ 8906.381162] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/07.t against ext4 filesystem
[ 8906.852345] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/07.t against lustre filesystem
[ 8907.074124] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/07.t against lustre filesystem
[ 8907.572038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8907.796476] Lustre: DEBUG MARKER: ext4 report
[ 8908.072978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8908.290483] Lustre: DEBUG MARKER: lustre report
[ 8909.690032] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_08: truncate returns EPERM if the named file has its immutable or append-only flag set ========================================================== 22:22:23 \(1764886943\)
[ 8909.915945] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_08: truncate returns EPERM if the named file has its immutable or append-only flag set ========================================================== 22:22:23 (1764886943)
[ 8910.189521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/08.t against ext4 filesystem
[ 8910.410659] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/08.t against ext4 filesystem
[ 8910.844957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/08.t against lustre filesystem
[ 8911.073286] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/08.t against lustre filesystem
[ 8911.522389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8911.739601] Lustre: DEBUG MARKER: ext4 report
[ 8912.019786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8912.245661] Lustre: DEBUG MARKER: lustre report
[ 8913.670127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_09: truncate returns EISDIR if the named file is a directory ========================================================== 22:22:27 \(1764886947\)
[ 8913.894432] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_09: truncate returns EISDIR if the named file is a directory ========================================================== 22:22:27 (1764886947)
[ 8914.166034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/09.t against ext4 filesystem
[ 8914.388420] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/09.t against ext4 filesystem
[ 8914.838601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/09.t against lustre filesystem
[ 8915.056123] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/09.t against lustre filesystem
[ 8915.533316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8915.752738] Lustre: DEBUG MARKER: ext4 report
[ 8916.033566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8916.257362] Lustre: DEBUG MARKER: lustre report
[ 8917.692660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_10: truncate returns EROFS if the named file resides on a read-only file system ========================================================== 22:22:31 \(1764886951\)
[ 8917.920225] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_10: truncate returns EROFS if the named file resides on a read-only file system ========================================================== 22:22:31 (1764886951)
[ 8918.197816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/10.t against ext4 filesystem
[ 8918.420169] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/10.t against ext4 filesystem
[ 8918.850871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/10.t against lustre filesystem
[ 8919.070329] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/10.t against lustre filesystem
[ 8919.520335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8919.731654] Lustre: DEBUG MARKER: ext4 report
[ 8920.020893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8920.237511] Lustre: DEBUG MARKER: lustre report
[ 8921.694470] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_11: truncate returns ETXTBSY the file is a pure procedure \(shared text\) file that is being executed ========================================================== 22:22:35 \(1764886955\)
[ 8921.915814] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_11: truncate returns ETXTBSY the file is a pure procedure (shared text) file that is being executed ========================================================== 22:22:35 (1764886955)
[ 8922.192837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/11.t against ext4 filesystem
[ 8922.410329] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/11.t against ext4 filesystem
[ 8922.842624] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/11.t against lustre filesystem
[ 8923.066884] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/11.t against lustre filesystem
[ 8923.508412] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8923.728118] Lustre: DEBUG MARKER: ext4 report
[ 8924.007657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8924.227330] Lustre: DEBUG MARKER: lustre report
[ 8925.650769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_12: truncate returns EFBIG or EINVAL if the length argument was greater than the maximum file size ========================================================== 22:22:39 \(1764886959\)
[ 8925.880455] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_12: truncate returns EFBIG or EINVAL if the length argument was greater than the maximum file size ========================================================== 22:22:39 (1764886959)
[ 8926.160284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/12.t against ext4 filesystem
[ 8926.374891] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/12.t against ext4 filesystem
[ 8926.834657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/12.t against lustre filesystem
[ 8927.057346] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/12.t against lustre filesystem
[ 8927.525545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8927.744928] Lustre: DEBUG MARKER: ext4 report
[ 8928.020870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8928.236463] Lustre: DEBUG MARKER: lustre report
[ 8929.647612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_13: ftruncate returns EINVAL if the length argument was less than 0 ========================================================== 22:22:43 \(1764886963\)
[ 8929.872370] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_13: ftruncate returns EINVAL if the length argument was less than 0 ========================================================== 22:22:43 (1764886963)
[ 8930.146504] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/13.t against ext4 filesystem
[ 8930.368549] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/13.t against ext4 filesystem
[ 8930.837230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/13.t against lustre filesystem
[ 8931.056206] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/13.t against lustre filesystem
[ 8931.533380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8931.758902] Lustre: DEBUG MARKER: ext4 report
[ 8932.031294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8932.250073] Lustre: DEBUG MARKER: lustre report
[ 8933.655536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test ftruncate_14: truncate returns EFAULT if the path argument points outside the process\'s allocated address space ========================================================== 22:22:47 \(1764886967\)
[ 8933.879358] Lustre: DEBUG MARKER: == run_pjdfstest test ftruncate_14: truncate returns EFAULT if the path argument points outside the process's allocated address space ========================================================== 22:22:47 (1764886967)
[ 8934.157995] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/14.t against ext4 filesystem
[ 8934.385021] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/14.t against ext4 filesystem
[ 8934.822764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/ftruncate\/14.t against lustre filesystem
[ 8935.043037] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/ftruncate/14.t against lustre filesystem
[ 8935.496608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8935.716307] Lustre: DEBUG MARKER: ext4 report
[ 8935.986919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8936.211100] Lustre: DEBUG MARKER: lustre report
[ 8937.635805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test granular_00: NFSv4 granular permissions checking - WRITE_DATA vs APPEND_DATA on directories ========================================================== 22:22:51 \(1764886971\)
[ 8937.859293] Lustre: DEBUG MARKER: == run_pjdfstest test granular_00: NFSv4 granular permissions checking - WRITE_DATA vs APPEND_DATA on directories ========================================================== 22:22:51 (1764886971)
[ 8938.133622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/00.t against ext4 filesystem
[ 8938.354531] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/00.t against ext4 filesystem
[ 8938.780006] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/00.t against lustre filesystem
[ 8938.998835] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/00.t against lustre filesystem
[ 8939.441476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8939.662299] Lustre: DEBUG MARKER: ext4 report
[ 8939.936968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8940.157516] Lustre: DEBUG MARKER: lustre report
[ 8941.556471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test granular_01: NFSv4 granular permissions checking - ACL_READ_ATTRIBUTES and ACL_WRITE_ATTRIBUTES ========================================================== 22:22:55 \(1764886975\)
[ 8941.781224] Lustre: DEBUG MARKER: == run_pjdfstest test granular_01: NFSv4 granular permissions checking - ACL_READ_ATTRIBUTES and ACL_WRITE_ATTRIBUTES ========================================================== 22:22:55 (1764886975)
[ 8942.056680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/01.t against ext4 filesystem
[ 8942.272890] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/01.t against ext4 filesystem
[ 8942.707628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/01.t against lustre filesystem
[ 8942.928080] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/01.t against lustre filesystem
[ 8943.369819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8943.604062] Lustre: DEBUG MARKER: ext4 report
[ 8943.878331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8944.098350] Lustre: DEBUG MARKER: lustre report
[ 8945.514256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test granular_02: NFSv4 granular permissions checking - ACL_READ_ACL and ACL_WRITE_ACL ========================================================== 22:22:59 \(1764886979\)
[ 8945.740323] Lustre: DEBUG MARKER: == run_pjdfstest test granular_02: NFSv4 granular permissions checking - ACL_READ_ACL and ACL_WRITE_ACL ========================================================== 22:22:59 (1764886979)
[ 8946.017538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/02.t against ext4 filesystem
[ 8946.236567] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/02.t against ext4 filesystem
[ 8946.667671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/02.t against lustre filesystem
[ 8946.889011] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/02.t against lustre filesystem
[ 8947.330053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8947.548496] Lustre: DEBUG MARKER: ext4 report
[ 8947.821855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8948.043822] Lustre: DEBUG MARKER: lustre report
[ 8949.492509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test granular_03: NFSv4 granular permissions checking - DELETE and DELETE_CHILD ========================================================== 22:23:03 \(1764886983\)
[ 8949.715241] Lustre: DEBUG MARKER: == run_pjdfstest test granular_03: NFSv4 granular permissions checking - DELETE and DELETE_CHILD ========================================================== 22:23:03 (1764886983)
[ 8949.988272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/03.t against ext4 filesystem
[ 8950.209666] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/03.t against ext4 filesystem
[ 8950.646054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/03.t against lustre filesystem
[ 8950.863297] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/03.t against lustre filesystem
[ 8951.308199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8951.529333] Lustre: DEBUG MARKER: ext4 report
[ 8951.802260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8952.024278] Lustre: DEBUG MARKER: lustre report
[ 8953.401049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test granular_04: NFSv4 granular permissions checking - ACL_WRITE_OWNER ========================================================== 22:23:07 \(1764886987\)
[ 8953.625755] Lustre: DEBUG MARKER: == run_pjdfstest test granular_04: NFSv4 granular permissions checking - ACL_WRITE_OWNER ========================================================== 22:23:07 (1764886987)
[ 8953.897880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/04.t against ext4 filesystem
[ 8954.117646] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/04.t against ext4 filesystem
[ 8954.545988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/04.t against lustre filesystem
[ 8954.771803] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/04.t against lustre filesystem
[ 8955.209925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8955.431672] Lustre: DEBUG MARKER: ext4 report
[ 8955.705805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8955.926119] Lustre: DEBUG MARKER: lustre report
[ 8957.375598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test granular_05: NFSv4 granular permissions checking - DELETE and DELETE_CHILD with directories ========================================================== 22:23:11 \(1764886991\)
[ 8957.602455] Lustre: DEBUG MARKER: == run_pjdfstest test granular_05: NFSv4 granular permissions checking - DELETE and DELETE_CHILD with directories ========================================================== 22:23:11 (1764886991)
[ 8957.878396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/05.t against ext4 filesystem
[ 8958.100099] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/05.t against ext4 filesystem
[ 8958.528993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/05.t against lustre filesystem
[ 8958.750966] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/05.t against lustre filesystem
[ 8959.197068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8959.416594] Lustre: DEBUG MARKER: ext4 report
[ 8959.693117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8959.915675] Lustre: DEBUG MARKER: lustre report
[ 8961.366823] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test granular_06: NFSv4 granular permissions checking - setuid and setgid are cleared when non-owner calls chown ========================================================== 22:23:15 \(1764886995\)
[ 8961.594779] Lustre: DEBUG MARKER: == run_pjdfstest test granular_06: NFSv4 granular permissions checking - setuid and setgid are cleared when non-owner calls chown ========================================================== 22:23:15 (1764886995)
[ 8961.878977] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/06.t against ext4 filesystem
[ 8962.098089] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/06.t against ext4 filesystem
[ 8962.531022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/granular\/06.t against lustre filesystem
[ 8962.752411] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/granular/06.t against lustre filesystem
[ 8963.203576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8963.427990] Lustre: DEBUG MARKER: ext4 report
[ 8963.703572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8963.920456] Lustre: DEBUG MARKER: lustre report
[ 8965.353813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test link_00: link creates hardlinks ==== 22:23:19 \(1764886999\)
[ 8965.577169] Lustre: DEBUG MARKER: == run_pjdfstest test link_00: link creates hardlinks ==== 22:23:19 (1764886999)
[ 8965.848125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/link\/00.t against ext4 filesystem
[ 8966.075151] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/link/00.t against ext4 filesystem
[ 8977.566250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/link\/00.t against lustre filesystem
[ 8977.792222] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/link/00.t against lustre filesystem
[ 8989.408464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8989.630011] Lustre: DEBUG MARKER: ext4 report
[ 8989.912302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8990.136034] Lustre: DEBUG MARKER: lustre report
[ 8991.603808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test link_01: link returns ENOTDIR if a component of either path prefix is not a directory ========================================================== 22:23:45 \(1764887025\)
[ 8991.830086] Lustre: DEBUG MARKER: == run_pjdfstest test link_01: link returns ENOTDIR if a component of either path prefix is not a directory ========================================================== 22:23:45 (1764887025)
[ 8992.110108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/link\/01.t against ext4 filesystem
[ 8992.337671] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/link/01.t against ext4 filesystem
[ 8992.960775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/link\/01.t against lustre filesystem
[ 8993.182210] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/link/01.t against lustre filesystem
[ 8993.825245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8994.042110] Lustre: DEBUG MARKER: ext4 report
[ 8994.318325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8994.542028] Lustre: DEBUG MARKER: lustre report
[ 8995.983866] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test link_02: link returns ENAMETOOLONG if a component of either pathname exceeded {NAME_MAX} characters ========================================================== 22:23:50 \(1764887030\)
[ 8996.204751] Lustre: DEBUG MARKER: == run_pjdfstest test link_02: link returns ENAMETOOLONG if a component of either pathname exceeded {NAME_MAX} characters ========================================================== 22:23:50 (1764887030)
[ 8996.485555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/link\/02.t against ext4 filesystem
[ 8996.703603] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/link/02.t against ext4 filesystem
[ 8997.275158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/link\/02.t against lustre filesystem
[ 8997.495435] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/link/02.t against lustre filesystem
[ 8998.096846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ext4 report
[ 8998.318702] Lustre: DEBUG MARKER: ext4 report
[ 8998.595205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lustre report
[ 8998.815945] Lustre: DEBUG MARKER: lustre report
[ 9000.244134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == run_pjdfstest test link_03: link returns ENAMETOOLONG if an entire length of either path name exceeded {PATH_MAX} characters ========================================================== 22:23:54 \(1764887034\)
[ 9000.470643] Lustre: DEBUG MARKER: == run_pjdfstest test link_03: link returns ENAMETOOLONG if an entire length of either path name exceeded {PATH_MAX} characters ========================================================== 22:23:54 (1764887034)
[ 9000.747685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/link\/03.t against ext4 filesystem
[ 9000.978628] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/link/03.t against ext4 filesystem
[ 9002.898841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Run \/usr\/share\/pjdfstest\/link\/03.t against lustre filesystem
[ 9003.117280] Lustre: DEBUG MARKER: Run /usr/share/pjdfstest/link/03.t against lustre filesystem
[ 9003.825189] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [lnet_discovery:181288]
[ 9003.825278] Modules linked in: mgc(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) lnet_selftest(OE) ksocklnd(OE) lnet(OE) libcfs(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common pcspkr i2c_piix4 virtio_balloon joydev drm fuse ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net virtio_blk ghash_clmulni_intel net_failover failover serio_raw [last unloaded: libcfs(OE)]
[ 9003.825536] CPU: 0 PID: 181288 Comm: lnet_discovery Kdump: loaded Tainted: G           OE     -------  ---  5.14.0-503.40.1.el9_5.x86_64 #1
[ 9003.825607] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 9003.825670] RIP: 0010:__raw_callee_save___pv_queued_spin_unlock+0xc/0x16
[ 9003.825778] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 52 b8 01 00 00 00 31 d2 f0 0f b0 17 <3c> 01 75 06 5a c3 cc cc cc cc 56 0f b6 f0 e8 a1 ff ff ff 5e 5a c3
[ 9003.825838] RSP: 0018:ffffb29740a07cf8 EFLAGS: 00000246
[ 9003.825901] RAX: 0000000000000001 RBX: ffff91fed1d96e00 RCX: 0000000000000000
[ 9003.825963] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: ffff91feb2837fc0
[ 9003.826026] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffffc0dcfd20
[ 9003.826082] R10: 0000000000000128 R11: ffffffff00000010 R12: ffff91fed3f518c0
[ 9003.826139] R13: ffff91fed1d96edc R14: 0000000000000000 R15: ffff91fed1d96e00
[ 9003.826196] FS:  0000000000000000(0000) GS:ffff91ffbbc00000(0000) knlGS:0000000000000000
[ 9003.826253] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9003.826310] CR2: 00007f1b4bbed000 CR3: 0000000109448001 CR4: 00000000000606f0
[ 9003.826369] Call Trace:
[ 9003.826427]  <IRQ>
[ 9003.826490]  ? show_trace_log_lvl+0x1c4/0x2df
[ 9003.826593]  ? show_trace_log_lvl+0x1c4/0x2df
[ 9003.826653]  ? _raw_spin_unlock+0xa/0x30
[ 9003.826713]  ? watchdog_timer_fn+0x1ad/0x210
[ 9003.826802]  ? __pfx_watchdog_timer_fn+0x10/0x10
[ 9003.826862]  ? __hrtimer_run_queues+0x112/0x2b0
[ 9003.826938]  ? hrtimer_interrupt+0xfc/0x210
[ 9003.826996]  ? kvm_sched_clock_read+0xd/0x20
[ 9003.827072]  ? __sysvec_apic_timer_interrupt+0x4e/0x100
[ 9003.827149]  ? sysvec_apic_timer_interrupt+0x6d/0x90
[ 9003.827208]  </IRQ>
[ 9003.827264]  <TASK>
[ 9003.827320]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[ 9003.827405]  ? __raw_callee_save___pv_queued_spin_unlock+0xc/0x16
[ 9003.827467]  _raw_spin_unlock+0xa/0x30
[ 9003.827532]  lnet_initiate_peer_discovery+0x155/0x3c0 [lnet]
[ 9003.827845]  lnet_select_pathway+0x218/0x620 [lnet]
[ 9003.827955]  lnet_send+0x6d/0x1e0 [lnet]
[ 9003.828068]  lnet_peer_discovery_complete+0x21c/0x390 [lnet]
[ 9003.828180]  lnet_peer_discovery+0x485/0xaf0 [lnet]
[ 9003.828291]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 9003.828370]  ? __pfx_lnet_peer_discovery+0x10/0x10 [lnet]
[ 9003.828478]  kthread+0xe0/0x100
[ 9003.828559]  ? __pfx_kthread+0x10/0x10
[ 9003.828621]  ret_from_fork+0x2c/0x50
[ 9003.828694]  </TASK>
[ 9003.828759] Kernel panic - not syncing: softlockup: hung tasks
[ 9003.879342] CPU: 0 PID: 181288 Comm: lnet_discovery Kdump: loaded Tainted: G           OEL    -------  ---  5.14.0-503.40.1.el9_5.x86_64 #1
[ 9003.881553] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 9003.882586] Call Trace:
[ 9003.883053]  <IRQ>
[ 9003.883432]  dump_stack_lvl+0x34/0x48
[ 9003.884124]  panic+0x107/0x2bb
[ 9003.884707]  watchdog_timer_fn.cold+0xc/0x16
[ 9003.885500]  ? __pfx_watchdog_timer_fn+0x10/0x10
[ 9003.886339]  __hrtimer_run_queues+0x112/0x2b0
[ 9003.887146]  hrtimer_interrupt+0xfc/0x210
[ 9003.887880]  ? kvm_sched_clock_read+0xd/0x20
[ 9003.888677]  __sysvec_apic_timer_interrupt+0x4e/0x100
[ 9003.889594]  sysvec_apic_timer_interrupt+0x6d/0x90
[ 9003.890470]  </IRQ>
[ 9003.890867]  <TASK>
[ 9003.891258]  asm_sysvec_apic_timer_interrupt+0x16/0x20
[ 9003.892182] RIP: 0010:__raw_callee_save___pv_queued_spin_unlock+0xc/0x16
[ 9003.893405] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 52 b8 01 00 00 00 31 d2 f0 0f b0 17 <3c> 01 75 06 5a c3 cc cc cc cc 56 0f b6 f0 e8 a1 ff ff ff 5e 5a c3
[ 9003.896707] RSP: 0018:ffffb29740a07cf8 EFLAGS: 00000246
[ 9003.897646] RAX: 0000000000000001 RBX: ffff91fed1d96e00 RCX: 0000000000000000
[ 9003.898919] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: ffff91feb2837fc0
[ 9003.900197] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffffc0dcfd20
[ 9003.901466] R10: 0000000000000128 R11: ffffffff00000010 R12: ffff91fed3f518c0
[ 9003.902753] R13: ffff91fed1d96edc R14: 0000000000000000 R15: ffff91fed1d96e00
[ 9003.904027]  _raw_spin_unlock+0xa/0x30
[ 9003.904714]  lnet_initiate_peer_discovery+0x155/0x3c0 [lnet]
[ 9003.905785]  lnet_select_pathway+0x218/0x620 [lnet]
[ 9003.906717]  lnet_send+0x6d/0x1e0 [lnet]
[ 9003.907480]  lnet_peer_discovery_complete+0x21c/0x390 [lnet]
[ 9003.908561]  lnet_peer_discovery+0x485/0xaf0 [lnet]
[ 9003.909492]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 9003.910432]  ? __pfx_lnet_peer_discovery+0x10/0x10 [lnet]
[ 9003.911444]  kthread+0xe0/0x100
[ 9003.912018]  ? __pfx_kthread+0x10/0x10
[ 9003.912701]  ret_from_fork+0x2c/0x50
[ 9003.913369]  </TASK>
