[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-327.22.2.el7_lustre.x86_64 (jenkins@onyx-1-sdg1-el7-x8664.onyx.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Jul 12 07:08:41 PDT 2016 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-327.22.2.el7_lustre.x86_64 root=UUID=c9bbe605-1ac5-4225-97fc-fef08cf8e8c3 ro crashkernel=auto console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] DMI: Red Hat KVM, BIOS 0.5.1 01/01/2007 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] AGP: No AGP bridge found [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000E0000000 mask 3FFFE0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT not supported by CPU. [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Base memory trampoline at [ffff880000094000] 94000 size 24576 [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x01f33000, 0x01f33fff] PGTABLE [ 0.000000] BRK [0x01f34000, 0x01f34fff] PGTABLE [ 0.000000] BRK [0x01f35000, 0x01f35fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x7fc00000-0x7fdfffff] [ 0.000000] [mem 0x7fc00000-0x7fdfffff] page 2M [ 0.000000] BRK [0x01f36000, 0x01f36fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x7c000000-0x7fbfffff] [ 0.000000] [mem 0x7c000000-0x7fbfffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x00100000-0x7bffffff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x7bffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x7fe00000-0x7fffcfff] [ 0.000000] [mem 0x7fe00000-0x7fffcfff] page 4k [ 0.000000] BRK [0x01f37000, 0x01f37fff] PGTABLE [ 0.000000] RAMDISK: [mem 0x35ece000-0x36f5efff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 161MB of memory at 688MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] [ffffea0000000000-ffffea0001ffffff] PMD -> [ffff88007d600000-ffff88007f5fffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] On node 0 totalpages: 524185 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 23 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 8128 pages used for memmap [ 0.000000] DMA32 zone: 520189 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 31 pages/cpu @ffff88007fc00000 s87232 r8192 d31552 u1048576 [ 0.000000] pcpu-alloc: s87232 r8192 d31552 u1048576 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0d240 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-327.22.2.el7_lustre.x86_64 root=UUID=c9bbe605-1ac5-4225-97fc-fef08cf8e8c3 ro crashkernel=auto console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] AGP: Checking aperture... [ 0.000000] AGP: No AGP bridge found [ 0.000000] Memory: 1865484k/2097140k available (6444k kernel code, 400k absent, 231256k reserved, 4262k data, 1624k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-1. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.748 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.49 BogoMIPS (lpj=2593748) [ 0.003015] pid_max: default: 32768 minimum: 301 [ 0.004020] Security Framework initialized [ 0.005017] SELinux: Initializing. [ 0.006011] SELinux: Starting in permissive mode [ 0.006185] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.008278] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.010105] Mount-cache hash table entries: 4096 [ 0.011140] Initializing cgroup subsys memory [ 0.012009] Initializing cgroup subsys devices [ 0.014002] Initializing cgroup subsys freezer [ 0.015003] Initializing cgroup subsys net_cls [ 0.016003] Initializing cgroup subsys blkio [ 0.017003] Initializing cgroup subsys perf_event [ 0.019003] Initializing cgroup subsys hugetlb [ 0.020067] Disabled fast string operations [ 0.021013] mce: CPU supports 10 MCE banks [ 0.022045] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 tlb_flushall_shift: 6 [ 0.024068] Freeing SMP alternatives: 28k freed [ 0.029562] ACPI: Core revision 20130517 [ 0.031500] ACPI: All ACPI Tables successfully acquired [ 0.032732] ftrace: allocating 24604 entries in 97 pages [ 0.041914] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.042013] smpboot: CPU0: Intel QEMU Virtual CPU version (cpu64-rhel6) (fam: 06, model: 0d, stepping: 03) [ 0.047000] Performance Events: Broken PMU hardware detected, using software events only. [ 0.048002] Failed to access perfctr msr (MSR c2 is 0) [ 0.050698] NMI watchdog: disabled (cpu0): hardware events not enabled [ 0.052002] smpboot: Booting Node 0, Processors #1 OK [ 0.050692] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.050692] Disabled fast string operations [ 0.069022] Brought up 2 CPUs [ 0.069013] kvm-stealtime: cpu 1, msr 7fd0d240 [ 0.070003] smpboot: Total of 2 processors activated (10374.99 BogoMIPS) [ 0.072466] devtmpfs: initialized [ 0.075792] EVM: security.selinux [ 0.076003] EVM: security.ima [ 0.077002] EVM: security.capability [ 0.079162] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.080081] NET: Registered protocol family 16 [ 0.082244] ACPI: bus type PCI registered [ 0.083004] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.084062] PCI: Using configuration type 1 for base access [ 0.086088] ACPI: Added _OSI(Module Device) [ 0.087002] ACPI: Added _OSI(Processor Device) [ 0.088002] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.089002] ACPI: Added _OSI(Processor Aggregator Device) [ 0.091371] ACPI: EC: Look up EC in DSDT [ 0.092861] ACPI: Interpreter enabled [ 0.093006] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20130517/hwxface-571) [ 0.096003] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130517/hwxface-571) [ 0.099003] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20130517/hwxface-571) [ 0.101003] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20130517/hwxface-571) [ 0.104006] ACPI: (supports S0 S5) [ 0.105002] ACPI: Using IOAPIC for interrupt routing [ 0.106013] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.110027] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.111006] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.112005] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.113015] acpi PNP0A03:00: host bridge window [io 0x0000-0x0cf7] (ignored) [ 0.113017] acpi PNP0A03:00: host bridge window [io 0x0d00-0xffff] (ignored) [ 0.113019] acpi PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff] (ignored) [ 0.113020] acpi PNP0A03:00: host bridge window [mem 0xe0000000-0xfebfffff] (ignored) [ 0.113022] PCI: root bus 00: using default resources [ 0.113025] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.114000] acpiphp: Slot [1] registered [ 0.114000] acpiphp: Slot [2] registered [ 0.114000] acpiphp: Slot [3] registered [ 0.114000] acpiphp: Slot [4] registered [ 0.114000] acpiphp: Slot [5] registered [ 0.114014] acpiphp: Slot [6] registered [ 0.115016] acpiphp: Slot [7] registered [ 0.116016] acpiphp: Slot [8] registered [ 0.117016] acpiphp: Slot [9] registered [ 0.118016] acpiphp: Slot [10] registered [ 0.119016] acpiphp: Slot [11] registered [ 0.120019] acpiphp: Slot [12] registered [ 0.121017] acpiphp: Slot [13] registered [ 0.122000] acpiphp: Slot [14] registered [ 0.122016] acpiphp: Slot [15] registered [ 0.123016] acpiphp: Slot [16] registered [ 0.124016] acpiphp: Slot [17] registered [ 0.126004] acpiphp: Slot [18] registered [ 0.127016] acpiphp: Slot [19] registered [ 0.128020] acpiphp: Slot [20] registered [ 0.129016] acpiphp: Slot [21] registered [ 0.130016] acpiphp: Slot [22] registered [ 0.131016] acpiphp: Slot [23] registered [ 0.132016] acpiphp: Slot [24] registered [ 0.133016] acpiphp: Slot [25] registered [ 0.134016] acpiphp: Slot [26] registered [ 0.135000] acpiphp: Slot [27] registered [ 0.135017] acpiphp: Slot [28] registered [ 0.136000] acpiphp: Slot [29] registered [ 0.136000] acpiphp: Slot [30] registered [ 0.136000] acpiphp: Slot [31] registered [ 0.136000] PCI host bridge to bus 0000:00 [ 0.136000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.136000] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] [ 0.136003] pci_bus 0000:00: root bus resource [mem 0x00000000-0x3fffffffffff] [ 0.137040] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.137302] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.138185] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.138366] pci 0000:00:01.1: reg 0x20: [io 0xc000-0xc00f] [ 0.138446] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.139002] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.140002] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.141000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.141113] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300 [ 0.141295] pci 0000:00:01.2: reg 0x20: [io 0xc020-0xc03f] [ 0.141476] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.141728] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.142009] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.143000] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000 [ 0.145007] pci 0000:00:02.0: reg 0x10: [mem 0xf0000000-0xf1ffffff pref] [ 0.147008] pci 0000:00:02.0: reg 0x14: [mem 0xf2000000-0xf2000fff] [ 0.162005] pci 0000:00:02.0: reg 0x30: [mem 0xf2010000-0xf201ffff pref] [ 0.162150] pci 0000:00:03.0: [10ec:8139] type 00 class 0x020000 [ 0.162252] pci 0000:00:03.0: reg 0x10: [io 0xc100-0xc1ff] [ 0.162318] pci 0000:00:03.0: reg 0x14: [mem 0xf2020000-0xf20200ff] [ 0.162630] pci 0000:00:03.0: reg 0x30: [mem 0xf2030000-0xf203ffff pref] [ 0.163019] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 [ 0.163102] pci 0000:00:04.0: reg 0x10: [io 0xc200-0xc23f] [ 0.163153] pci 0000:00:04.0: reg 0x14: [mem 0xf2040000-0xf2040fff] [ 0.163645] pci 0000:00:05.0: [1af4:1002] type 00 class 0x050000 [ 0.163698] pci 0000:00:05.0: reg 0x10: [io 0xc240-0xc25f] [ 0.164323] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.166066] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.167860] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.169062] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.171034] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.173192] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.174081] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.175002] vgaarb: loaded [ 0.176002] vgaarb: bridge control possible 0000:00:02.0 [ 0.177055] SCSI subsystem initialized [ 0.178026] ACPI: bus type USB registered [ 0.179023] usbcore: registered new interface driver usbfs [ 0.180000] usbcore: registered new interface driver hub [ 0.180011] usbcore: registered new device driver usb [ 0.182062] PCI: Using ACPI for IRQ routing [ 0.183002] PCI: pci_cache_line_size set to 64 bytes [ 0.183098] e820: reserve RAM buffer [mem 0x0009dc00-0x0009ffff] [ 0.183100] e820: reserve RAM buffer [mem 0x7fffd000-0x7fffffff] [ 0.183200] NetLabel: Initializing [ 0.184002] NetLabel: domain hash size = 128 [ 0.185001] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.186000] NetLabel: unlabeled traffic allowed by default [ 0.186000] Switching to clocksource kvm-clock [ 0.192867] pnp: PnP ACPI init [ 0.193981] ACPI: bus type PNP registered [ 0.195255] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.195312] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.195350] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.195379] pnp 00:03: [dma 2] [ 0.195389] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.195467] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.195586] pnp: PnP ACPI: found 5 devices [ 0.196877] ACPI: bus type PNP unregistered [ 0.203586] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] [ 0.203588] pci_bus 0000:00: resource 5 [mem 0x00000000-0x3fffffffffff] [ 0.203618] NET: Registered protocol family 2 [ 0.205066] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.207368] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.208924] TCP: Hash tables configured (established 16384 bind 16384) [ 0.210478] TCP: reno registered [ 0.211591] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.213036] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.214583] NET: Registered protocol family 1 [ 0.215840] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.217269] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.218693] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.220391] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.222072] pci 0000:00:02.0: Boot video device [ 0.222096] PCI: CLS 0 bytes, default 64 [ 0.222153] Unpacking initramfs... [ 0.442684] Freeing initrd memory: 16964k freed [ 0.449350] microcode: CPU0 sig=0x6d3, pf=0x1, revision=0x1 [ 0.450872] microcode: CPU1 sig=0x6d3, pf=0x1, revision=0x1 [ 0.452340] microcode: Microcode Update Driver: v2.00 , Peter Oruba [ 0.455040] sha1_ssse3: Neither AVX nor AVX2 nor SSSE3 is available/usable. [ 0.456620] sha256_ssse3: Neither AVX nor SSSE3 is available/usable. [ 0.458257] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.459737] Initialise system trusted keyring [ 0.461106] audit: initializing netlink socket (disabled) [ 0.462495] type=2000 audit(1469497346.437:1): initialized [ 0.481713] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.484116] zpool: loaded [ 0.485136] zbud: loaded [ 0.486339] VFS: Disk quotas dquot_6.5.2 [ 0.487543] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.489200] msgmni has been set to 3676 [ 0.490421] Key type big_key registered [ 0.491589] SELinux: Registering netfilter hooks [ 0.493135] NET: Registered protocol family 38 [ 0.495154] Key type asymmetric registered [ 0.497089] Asymmetric key parser 'x509' registered [ 0.499423] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.501726] io scheduler noop registered [ 0.502950] io scheduler deadline registered (default) [ 0.504301] io scheduler cfq registered [ 0.505552] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.506904] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.508863] intel_idle: does not run on family 6 model 13 [ 0.508969] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.514958] ACPI: Power Button [PWRF] [ 0.518452] GHES: HEST is not enabled! [ 0.520214] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.542872] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.544487] Non-volatile memory driver v1.3 [ 0.545655] Linux agpgart interface v0.103 [ 0.546895] crash memory driver: version 1.1 [ 0.548078] rdac: device handler registered [ 0.549407] hp_sw: device handler registered [ 0.550637] emc: device handler registered [ 0.551824] alua: device handler registered [ 0.553072] libphy: Fixed MDIO Bus: probed [ 0.554305] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.555755] ehci-pci: EHCI PCI platform driver [ 0.556955] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.558364] ohci-pci: OHCI PCI platform driver [ 0.559586] uhci_hcd: USB Universal Host Controller Interface driver [ 0.561294] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.562628] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.564818] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.566157] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.567578] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.569160] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.571502] usb usb1: Product: UHCI Host Controller [ 0.572896] usb usb1: Manufacturer: Linux 3.10.0-327.22.2.el7_lustre.x86_64 uhci_hcd [ 0.575221] usb usb1: SerialNumber: 0000:00:01.2 [ 0.576638] hub 1-0:1.0: USB hub found [ 0.577884] hub 1-0:1.0: 2 ports detected [ 0.579278] usbcore: registered new interface driver usbserial [ 0.580784] usbcore: registered new interface driver usbserial_generic [ 0.582376] usbserial: USB Serial support registered for generic [ 0.583901] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.586816] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.588202] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.589682] mousedev: PS/2 mouse device common for all mice [ 0.591354] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.594062] rtc_cmos 00:00: RTC can wake from S4 [ 0.596000] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.597510] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.599056] cpuidle: using governor menu [ 0.600318] hidraw: raw HID events driver (C) Jiri Kosina [ 0.601787] usbcore: registered new interface driver usbhid [ 0.603171] usbhid: USB HID core driver [ 0.604361] drop_monitor: Initializing network drop monitor service [ 0.605955] TCP: cubic registered [ 0.607081] Initializing XFRM netlink socket [ 0.608382] NET: Registered protocol family 10 [ 0.609914] NET: Registered protocol family 17 [ 0.611534] Loading compiled-in X.509 certificates [ 0.613043] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.616721] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.621391] Loaded X.509 cert 'CentOS Linux kernel signing key: da4766b66c80426fbd2a108d2710cb13a76ac649' [ 0.624054] registered taskstats version 1 [ 0.627883] Key type trusted registered [ 0.630975] Key type encrypted registered [ 0.633959] IMA: No TPM chip found, activating TPM-bypass! [ 0.635732] rtc_cmos 00:00: setting system clock to 2016-07-26 01:42:26 UTC (1469497346) [ 0.638864] Freeing unused kernel memory: 1624k freed [ 0.643433] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.647854] systemd[1]: Detected virtualization kvm. [ 0.649237] systemd[1]: Detected architecture x86-64. [ 0.650636] systemd[1]: Running in initial RAM disk. [ 0.652991] systemd[1]: Set hostname to . [ 0.696667] systemd[1]: Created slice -.slice. [ 0.697912] systemd[1]: Starting -.slice. [ 0.699902] systemd[1]: Listening on udev Control Socket. [ 0.701307] systemd[1]: Starting udev Control Socket. [ 0.703220] systemd[1]: Reached target Timers. [ 0.704480] systemd[1]: Starting Timers. [ 0.706311] systemd[1]: Created slice System Slice. [ 0.707649] systemd[1]: Starting System Slice. [ 0.709438] systemd[1]: Reached target Slices. [ 0.710685] systemd[1]: Starting Slices. [ 0.712521] systemd[1]: Reached target Local File Systems. [ 0.713931] systemd[1]: Starting Local File Systems. [ 0.716043] systemd[1]: Listening on udev Kernel Socket. [ 0.717405] systemd[1]: Starting udev Kernel Socket. [ 0.719447] systemd[1]: Listening on Journal Socket. [ 0.720706] systemd[1]: Starting Journal Socket. [ 0.722338] systemd[1]: Starting Journal Service... [ 0.726184] systemd[1]: Reached target Sockets. [ 0.727983] systemd[1]: Starting Sockets. [ 0.729559] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 0.747899] systemd[1]: Started dracut ask for additional cmdline parameters. [ 0.750033] systemd[1]: Starting dracut cmdline hook... [ 0.752342] systemd[1]: Starting Load Kernel Modules... [ 0.754606] systemd[1]: Starting Setup Virtual Console... [ 0.757173] systemd[1]: Reached target Swap. [ 0.758662] systemd[1]: Starting Swap. [ 0.761615] systemd[1]: Started Journal Service. [ 0.935095] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 0.937042] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 0.941523] FDC 0 is a S82078B [ 0.948319] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:4e:a8:ed, IRQ 10 [ 0.954162] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 0.959375] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.960384] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 0.963387] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 0.965299] libata version 3.00 loaded. [ 0.966263] [drm] Initialized drm 1.1.0 20060810 [ 0.967702] ata_piix 0000:00:01.1: version 2.13 [ 0.976806] scsi host0: ata_piix [ 0.978341] virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X [ 0.978368] virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X [ 1.002365] scsi host1: ata_piix [ 1.003502] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.005055] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.007039] [TTM] Zone kernel: Available graphics memory: 942050 kiB [ 1.007039] [TTM] Initializing pool allocator [ 1.007041] [TTM] Initializing DMA pool allocator [ 1.077520] [drm] fb mappable at 0xF0000000 [ 1.078759] [drm] vram aper at 0xF0000000 [ 1.078760] [drm] size 33554432 [ 1.078760] [drm] fb depth is 24 [ 1.078760] [drm] pitch is 3072 [ 1.082177] vda: vda1 vda2 vda4 [ 1.087111] fbcon: cirrusdrmfb (fb0) is primary device [ 1.117403] Console: switching to colour frame buffer device 128x48 [ 1.129654] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 1.129655] cirrus 0000:00:02.0: registered panic notifier [ 1.130733] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 1.287915] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.290169] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ 1.451158] tsc: Refined TSC clocksource calibration: 2593.670 MHz [ 1.507770] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 1.565587] systemd-journald[93]: Received SIGTERM from PID 1 (systemd). [ 1.605268] SELinux: Disabled at runtime. [ 1.606605] SELinux: Unregistering netfilter hooks [ 1.625049] type=1404 audit(1469497347.489:2): selinux=0 auid=4294967295 ses=4294967295 [ 1.635748] ip_tables: (C) 2000-2006 Netfilter Core Team [ 1.637221] systemd[1]: Inserted module 'ip_tables' [ 1.719574] RPC: Registered named UNIX socket transport module. [ 1.719575] RPC: Registered udp transport module. [ 1.719575] RPC: Registered tcp transport module. [ 1.719576] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.882075] spl: module verification failed: signature and/or required key missing - tainting kernel [ 1.887457] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 1.888381] SPL: Loaded module v0.6.5.7-1 [ 1.892114] znvpair: module license 'CDDL' taints kernel. [ 1.892114] Disabling lock debugging due to kernel taint [ 1.912711] ZFS: Loaded module v0.6.5.7-1, ZFS pool version 5000, ZFS filesystem version 5 [ 1.977182] EXT4-fs (vda1): re-mounted. Opts: (null) [ 2.002331] systemd-journald[352]: Received request to flush runtime journal from PID 1 [ 2.062051] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 2.104535] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 2.141679] ppdev: user-space parallel port driver [ 2.145487] Adding 2098172k swap on /dev/vda2. Priority:-1 extents:1 across:2098172k FS [ 2.243563] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 2.262960] Loading iSCSI transport class v2.0-870. [ 2.271254] iscsi: registered transport (iser) [ 2.306199] type=1305 audit(1469497348.170:3): audit_pid=590 old=0 auid=4294967295 ses=4294967295 res=1 [ 2.308412] RPC: Registered rdma transport module. [ 2.417450] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 2.423615] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 8.680704] FS-Cache: Loaded [ 8.695756] FS-Cache: Netfs 'nfs' registered for caching [ 8.702467] Key type dns_resolver registered [ 8.716441] NFS: Registering the id_resolver key type [ 8.718433] Key type id_resolver registered [ 8.720216] Key type id_legacy registered [ 127.686681] device-mapper: uevent: version 1.0.3 [ 127.687378] device-mapper: ioctl: 4.33.0-ioctl (2015-8-18) initialised: dm-devel@redhat.com [ 157.092890] SPL: using hostid 0x020a5405 [ 187.224250] LNet: HW CPU cores: 2, npartitions: 1 [ 187.238576] alg: No test for adler32 (adler32-zlib) [ 187.239389] alg: No test for crc32 (crc32-table) [ 195.270931] Lustre: Lustre: Build Version: 2.8.55_105_ga352071 [ 195.290065] LNet: Added LNI 10.2.5.84@tcp [8/256/0/180] [ 195.290825] LNet: Accept all, port 7988 [ 195.326872] Lustre: MGS: Connection restored to f14df7fe-c07f-3b6c-2e32-0b0fa5cf5034 (at 0@lo) [ 195.368681] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripecount in log lustre-MDT0000 [ 196.576175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 196.591223] Lustre: lustre-MDT0000: new disk, initializing [ 216.557474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.099542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 219.265883] Lustre: lustre-MDT0000: Connection restored to f14df7fe-c07f-3b6c-2e32-0b0fa5cf5034 (at 0@lo) [ 219.437653] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 220.019017] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 220.309993] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 220.619484] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 220.939221] Lustre: DEBUG MARKER: sync; sync; sync [ 221.789030] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 222.099465] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 234.579220] Lustre: MGS: Connection restored to a145d32a-5cd1-41e6-7d0b-221a323b762f (at 10.2.5.85@tcp) [ 248.475565] Lustre: lustre-MDT0000: Connection restored to a145d32a-5cd1-41e6-7d0b-221a323b762f (at 10.2.5.85@tcp) [ 248.477939] Lustre: Skipped 1 previous similar message [ 252.859780] Lustre: MGS: Connection restored to 63beb10b-1bf2-63ca-13cd-729944405c6c (at 10.2.5.86@tcp) [ 262.909426] Lustre: MGS: Connection restored to ce2cb6eb-d871-e5eb-b905-b6153cc825c1 (at 10.2.5.87@tcp) [ 262.911608] Lustre: Skipped 1 previous similar message [ 265.380629] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 265.749574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 265.944511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.130800] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 266.530915] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [ 276.076645] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 276.701133] Lustre: DEBUG MARKER: uname -n [ 277.700110] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 278.054574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 278.620104] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Client: Lustre version: 2.8.55_105_ga352071 [ 278.788335] Lustre: DEBUG MARKER: Client: Lustre version: 2.8.55_105_ga352071 [ 278.980597] Lustre: DEBUG MARKER: /usr/sbin/lctl lustre_build_version [ 279.330205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MDS: Lustre version: 2.8.55_105_ga352071 [ 279.545360] Lustre: DEBUG MARKER: MDS: Lustre version: 2.8.55_105_ga352071 [ 280.137020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OSS: Lustre version: 2.8.55_105_ga352071 [ 280.333639] Lustre: DEBUG MARKER: OSS: Lustre version: 2.8.55_105_ga352071 [ 280.604388] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: runtests ============----- Mon Jul 25 18:47:07 PDT 2016 [ 280.816350] Lustre: DEBUG MARKER: -----============= acceptance-small: runtests ============----- Mon Jul 25 18:47:07 PDT 2016 [ 281.170572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test complete, duration -o sec =========================================================== 18:47:07 \(1469497627\) [ 281.331516] Lustre: DEBUG MARKER: == runtests test complete, duration -o sec =========================================================== 18:47:07 (1469497627) [ 283.158702] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 283.550825] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 284.509820] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 284.815853] Lustre: DEBUG MARKER: cat /proc/mounts [ 286.661222] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 287.055244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 287.241208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.410939] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 287.784968] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 288.359691] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 288.699210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test 1: All Runtests ===================================================================== 18:47:15 \(1469497635\) [ 288.880203] Lustre: DEBUG MARKER: == runtests test 1: All Runtests ===================================================================== 18:47:15 (1469497635) [ 289.161106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark touching \/mnt\/lustre at Mon Jul 25 18:47:15 PDT 2016 \(@1469497635\) [ 289.350861] Lustre: DEBUG MARKER: touching /mnt/lustre at Mon Jul 25 18:47:15 PDT 2016 (@1469497635) [ 289.626817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create an empty file \/mnt\/lustre\/hosts.9944 [ 289.841070] Lustre: DEBUG MARKER: create an empty file /mnt/lustre/hosts.9944 [ 290.036790] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.9944 [ 290.204129] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.9944 [ 290.489586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing \/etc\/hosts and \/mnt\/lustre\/hosts.9944 [ 290.690857] Lustre: DEBUG MARKER: comparing /etc/hosts and /mnt/lustre/hosts.9944 [ 291.018137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark renaming \/mnt\/lustre\/hosts.9944 to \/mnt\/lustre\/hosts.9944.ren [ 291.213903] Lustre: DEBUG MARKER: renaming /mnt/lustre/hosts.9944 to /mnt/lustre/hosts.9944.ren [ 291.460439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.9944 again [ 291.655559] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.9944 again [ 291.890603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.9944 [ 292.072569] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.9944 [ 292.341071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/hosts.9944 [ 292.529790] Lustre: DEBUG MARKER: removing /mnt/lustre/hosts.9944 [ 292.791085] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.9944.2 [ 293.000956] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.9944.2 [ 293.260573] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.9944.2 to 123 bytes [ 293.444076] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.9944.2 to 123 bytes [ 293.657214] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating \/mnt\/lustre\/d1.runtests [ 293.838711] Lustre: DEBUG MARKER: creating /mnt/lustre/d1.runtests [ 295.009730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying 509 files from \/etc \/bin to \/mnt\/lustre\/d1.runtests\/etc \/bin at Mon Jul 25 18:47:21 PDT 2016 [ 295.200494] Lustre: DEBUG MARKER: copying 509 files from /etc /bin to /mnt/lustre/d1.runtests/etc /bin at Mon Jul 25 18:47:21 PDT 2016 [ 304.279155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing 509 newly copied files at Mon Jul 25 18:47:30 PDT 2016 [ 304.492164] Lustre: DEBUG MARKER: comparing 509 newly copied files at Mon Jul 25 18:47:30 PDT 2016 [ 308.116623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark finished at Mon Jul 25 18:47:34 PDT 2016 \(19\) [ 308.293682] Lustre: DEBUG MARKER: finished at Mon Jul 25 18:47:34 PDT 2016 (19) [ 309.429943] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 309.731044] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 311.071205] LustreError: 7954:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004c2e5500 x1540879853562560/t0(0) o13->lustre-OST0001-osc-MDT0000@10.2.5.85@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 313.486506] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.85@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.486513] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.85@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.920123] Lustre: 12010:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469497657/real 1469497657] req@ffff88004c9a3c00 x1540879853562624/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469497663 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 317.952227] Lustre: server umount lustre-MDT0000 complete [ 318.109642] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 318.430304] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 331.169616] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 331.532654] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 332.719025] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 333.040075] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 336.360111] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 336.649810] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 336.850307] Lustre: MGS: Connection restored to f14df7fe-c07f-3b6c-2e32-0b0fa5cf5034 (at 0@lo) [ 336.856365] Lustre: Skipped 1 previous similar message [ 341.178504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 341.470666] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 342.040512] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 342.330454] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 342.619700] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 342.919930] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 343.149196] Lustre: 7953:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469497684/real 1469497684] req@ffff88004c2e5b00 x1540879853562784/t0(0) o8->lustre-OST0001-osc-MDT0000@10.2.5.85@tcp:28/4 lens 520/544 e 0 to 1 dl 1469497689 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 343.235102] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 362.831129] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 363.189554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 363.384675] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 363.531077] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 368.971778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing 509 previously copied files [ 369.174925] Lustre: DEBUG MARKER: comparing 509 previously copied files [ 376.579826] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 376.897097] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 378.191174] LustreError: 7954:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880052e8f000 x1540879853565520/t0(0) o13->lustre-OST0001-osc-MDT0000@10.2.5.85@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 378.882497] Lustre: lustre-MDT0000: Not available for connect from 10.2.5.85@tcp (stopping) [ 383.881411] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.85@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.881530] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.85@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.890889] LustreError: Skipped 1 previous similar message [ 389.091118] Lustre: 14735:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469497728/real 1469497728] req@ffff880052db0000 x1540879853565600/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469497734 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 389.103087] Lustre: 14735:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 389.752069] Lustre: server umount lustre-MDT0000 complete [ 389.918513] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 390.249562] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 407.230927] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 407.559854] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 408.610570] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 408.951089] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 410.310589] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 411.350599] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 411.561333] Lustre: MGS: Connection restored to f14df7fe-c07f-3b6c-2e32-0b0fa5cf5034 (at 0@lo) [ 411.565211] Lustre: Skipped 8 previous similar messages [ 417.677229] Lustre: 7953:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469497758/real 1469497758] req@ffff880052db0f00 x1540879853565760/t0(0) o8->lustre-OST0001-osc-MDT0000@10.2.5.85@tcp:28/4 lens 520/544 e 0 to 1 dl 1469497763 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 420.438129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 420.730120] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 421.340129] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 421.649930] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 421.951104] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 422.250463] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 422.550134] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 432.677212] Lustre: 7953:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469497768/real 1469497768] req@ffff880052e8c300 x1540879853566080/t0(0) o8->lustre-OST0000-osc-MDT0000@10.2.5.85@tcp:28/4 lens 520/544 e 0 to 1 dl 1469497778 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 432.685675] Lustre: 7953:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 438.891140] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 439.230947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 439.451091] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 439.620201] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 444.987849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/d1.runtests [ 445.175170] Lustre: DEBUG MARKER: removing /mnt/lustre/d1.runtests [ 448.240732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark renaming \/mnt\/lustre\/hosts.9944.ren to \/mnt\/lustre\/hosts.9944 [ 448.469594] Lustre: DEBUG MARKER: renaming /mnt/lustre/hosts.9944.ren to /mnt/lustre/hosts.9944 [ 448.679819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.9944 [ 448.901016] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.9944 [ 450.581681] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/hosts.9944 [ 450.800141] Lustre: DEBUG MARKER: removing /mnt/lustre/hosts.9944 [ 451.078366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark verifying \/mnt\/lustre\/hosts.9944.2 is 123 bytes [ 451.281340] Lustre: DEBUG MARKER: verifying /mnt/lustre/hosts.9944.2 is 123 bytes [ 451.572153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark running createmany -d \/mnt\/lustre\/d1.runtests\/d 100 [ 451.759416] Lustre: DEBUG MARKER: running createmany -d /mnt/lustre/d1.runtests/d 100 [ 452.184367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark running unlinkmany -d \/mnt\/lustre\/d1.runtests\/d 100 [ 452.360532] Lustre: DEBUG MARKER: running unlinkmany -d /mnt/lustre/d1.runtests/d 100 [ 452.916006] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done [ 453.093467] Lustre: DEBUG MARKER: done [ 453.281656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 453.600103] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 455.301380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 456.660996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 458.031265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 464.621349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Space was freed: now 3584kB, was 3584kB. [ 464.809159] Lustre: DEBUG MARKER: Space was freed: now 3584kB, was 3584kB. [ 465.009403] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 465.399332] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 465.802027] Lustre: DEBUG MARKER: dmesg [ 466.460798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test complete, duration 186 sec ========================================================== 18:50:13 \(1469497813\) [ 466.662802] Lustre: DEBUG MARKER: == runtests test complete, duration 186 sec ========================================================== 18:50:13 (1469497813) [ 467.004946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity ============----- Mon Jul 25 18:50:13 PDT 2016 [ 467.171160] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Mon Jul 25 18:50:13 PDT 2016 [ 467.510567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test complete, duration -o sec ============================================================= 18:50:14 \(1469497814\) [ 467.704020] Lustre: DEBUG MARKER: == sanity test complete, duration -o sec ============================================================= 18:50:14 (1469497814) [ 469.390837] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 469.770038] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 470.770545] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 471.079796] Lustre: DEBUG MARKER: cat /proc/mounts [ 472.860994] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 473.241459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 473.445611] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 473.639831] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 474.066315] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 474.659173] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 476.290017] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 76 101g 42a 42b 42c 42d 45 68b 65ic 180 184c [ 476.452654] Lustre: DEBUG MARKER: excepting tests: 76 101g 42a 42b 42c 42d 45 68b 65ic 180 184c [ 476.680930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o 51b 51ba [ 476.876578] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o 51b 51ba [ 477.587198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0a: touch\; rm ========================================================================= 18:50:24 \(1469497824\) [ 477.799648] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 18:50:24 (1469497824) [ 478.006505] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 478.430801] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 478.829482] Lustre: DEBUG MARKER: dmesg [ 479.353315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0b: chmod 0755 \/mnt\/lustre ============================================================ 18:50:26 \(1469497826\) [ 479.588294] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 18:50:26 (1469497826) [ 479.840821] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 480.289602] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 480.704695] Lustre: DEBUG MARKER: dmesg [ 481.237845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0c: check import proc ================================================================= 18:50:27 \(1469497827\) [ 481.425694] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 18:50:27 (1469497827) [ 481.642643] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 482.081194] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 482.477942] Lustre: DEBUG MARKER: dmesg [ 482.923274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 1: mkdir\; remkdir\; rmdir ============================================================== 18:50:29 \(1469497829\) [ 483.107174] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ============================================================== 18:50:29 (1469497829) [ 483.387186] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 483.811117] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 484.248310] Lustre: DEBUG MARKER: dmesg [ 484.740818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 2: mkdir\; touch\; rmdir\; check file ==================================================== 18:50:31 \(1469497831\) [ 484.958831] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ==================================================== 18:50:31 (1469497831) [ 485.241791] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 485.652701] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 486.121844] Lustre: DEBUG MARKER: dmesg [ 486.599125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 3: mkdir\; touch\; rmdir\; check dir ===================================================== 18:50:33 \(1469497833\) [ 486.786188] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ===================================================== 18:50:33 (1469497833) [ 487.070996] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 487.563482] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 487.961718] Lustre: DEBUG MARKER: dmesg [ 488.410101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 4: mkdir\; touch dir\/file\; rmdir\; checkdir \(expect error\) ============================== 18:50:35 \(1469497835\) [ 488.613081] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ============================== 18:50:35 (1469497835) [ 488.870911] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 489.231643] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 489.658670] Lustre: DEBUG MARKER: dmesg [ 490.183183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 5: mkdir ...\/d5 ...\/d5\/d2\; chmod ...\/d5\/d2 ============================================ 18:50:36 \(1469497836\) [ 490.396836] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 18:50:36 (1469497836) [ 492.885778] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 493.288916] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 493.738882] Lustre: DEBUG MARKER: dmesg [ 494.226432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6a: touch f6a\; chmod f6a\; runas -u 500 -g 500 chmod f6a \(should return error\) ========= 18:50:41 \(1469497841\) [ 494.412985] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ========= 18:50:41 (1469497841) [ 494.658763] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 495.089644] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 495.556920] Lustre: DEBUG MARKER: dmesg [ 496.020152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6c: touch f6c\; chown f6c\; runas -u 500 -g 500 chown f6c \(should return error\) ========= 18:50:42 \(1469497842\) [ 496.225367] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ========= 18:50:42 (1469497842) [ 496.500432] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 496.925099] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 497.373646] Lustre: DEBUG MARKER: dmesg [ 497.872978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6e: touch f6e\; chgrp f6e\; runas -u 500 -g 500 chgrp f6e \(should return error\) ========= 18:50:44 \(1469497844\) [ 498.121024] Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ========= 18:50:44 (1469497844) [ 498.434695] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 498.807912] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 499.188515] Lustre: DEBUG MARKER: dmesg [ 499.649170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 18:50:46 \(1469497846\) [ 499.861422] Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 18:50:46 (1469497846) [ 501.699050] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 502.113029] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 502.586234] Lustre: DEBUG MARKER: dmesg [ 503.069555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 ...\/f6h \(should return error\) ================ 18:50:49 \(1469497849\) [ 503.237560] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) ================ 18:50:49 (1469497849) [ 503.548086] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 503.957833] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 504.384316] Lustre: DEBUG MARKER: dmesg [ 504.840467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 7a: mkdir ...\/d7\; mcreate ...\/d7\/f\; chmod ...\/d7\/f ==================================== 18:50:51 \(1469497851\) [ 505.017174] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 18:50:51 (1469497851) [ 505.286708] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 505.691184] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 506.099708] Lustre: DEBUG MARKER: dmesg [ 506.533958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 7b: mkdir ...\/d7\; mcreate d7\/f2\; echo foo \> d7\/f2 ===================================== 18:50:53 \(1469497853\) [ 506.742377] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 18:50:53 (1469497853) [ 507.028634] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 507.470277] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 507.890486] Lustre: DEBUG MARKER: dmesg [ 508.360814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 8: mkdir ...\/d8\; touch ...\/d8\/f\; chmod ...\/d8\/f ======================================= 18:50:55 \(1469497855\) [ 508.602161] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 18:50:55 (1469497855) [ 508.880908] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 509.287289] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 509.671315] Lustre: DEBUG MARKER: dmesg [ 510.177998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 9: mkdir ...\/d9 ...\/d9\/d2 ...\/d9\/d2\/d3 ================================================ 18:50:56 \(1469497856\) [ 510.410182] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 18:50:56 (1469497856) [ 510.661639] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 511.115491] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 511.532245] Lustre: DEBUG MARKER: dmesg [ 511.997346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 10: mkdir ...\/d10 ...\/d10\/d2\; touch ...\/d10\/d2\/f ====================================== 18:50:58 \(1469497858\) [ 512.220723] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 18:50:58 (1469497858) [ 512.520666] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 512.933335] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 513.362963] Lustre: DEBUG MARKER: dmesg [ 513.836885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 11: mkdir ...\/d11 d11\/d2\; chmod ...\/d11\/d2 ============================================ 18:51:00 \(1469497860\) [ 514.051106] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 18:51:00 (1469497860) [ 517.643740] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 518.087807] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 518.520189] Lustre: DEBUG MARKER: dmesg [ 519.040612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 12: touch ...\/d12\/f\; chmod ...\/d12\/f ...\/d12\/f ======================================== 18:51:05 \(1469497865\) [ 519.245555] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 18:51:05 (1469497865) [ 519.560788] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 519.968224] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 520.364806] Lustre: DEBUG MARKER: dmesg [ 520.820161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 13: creat ...\/d13\/f\; dd ...\/d13\/f\; \> ...\/d13\/f ======================================== 18:51:07 \(1469497867\) [ 521.032402] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 18:51:07 (1469497867) [ 521.310635] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 521.729513] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 522.143433] Lustre: DEBUG MARKER: dmesg [ 522.588512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 14: touch ...\/d14\/f\; rm ...\/d14\/f\; rm ...\/d14\/f ======================================= 18:51:09 \(1469497869\) [ 522.811507] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 18:51:09 (1469497869) [ 523.076323] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 523.454697] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 523.820642] Lustre: DEBUG MARKER: dmesg [ 524.312798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 15: touch ...\/d15\/f\; mv ...\/d15\/f ...\/d15\/f2 ========================================== 18:51:11 \(1469497871\) [ 524.543974] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 18:51:11 (1469497871) [ 524.830057] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 525.256487] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 525.701374] Lustre: DEBUG MARKER: dmesg [ 526.209649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 16: touch ...\/d16\/f\; rm -rf ...\/d16\/f ================================================= 18:51:12 \(1469497872\) [ 526.383080] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 18:51:12 (1469497872) [ 526.660480] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 527.098313] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 527.541398] Lustre: DEBUG MARKER: dmesg [ 528.070256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17a: symlinks: create, remove \(real\) ================================================== 18:51:14 \(1469497874\) [ 528.265711] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 18:51:14 (1469497874) [ 528.560996] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 528.971745] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 529.394232] Lustre: DEBUG MARKER: dmesg [ 530.050844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17b: symlinks: create, remove \(dangling\) ============================================== 18:51:16 \(1469497876\) [ 530.225750] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 18:51:16 (1469497876) [ 530.528764] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 530.940497] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 531.380395] Lustre: DEBUG MARKER: dmesg [ 531.851994] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17c: symlinks: open dangling \(should return error\) ==================================== 18:51:18 \(1469497878\) [ 532.049294] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 18:51:18 (1469497878) [ 532.313998] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 532.754456] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 533.124780] Lustre: DEBUG MARKER: dmesg [ 533.630615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17d: symlinks: create dangling ======================================================== 18:51:20 \(1469497880\) [ 533.833909] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 18:51:20 (1469497880) [ 534.124185] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 534.581657] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 535.009816] Lustre: DEBUG MARKER: dmesg [ 535.531275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17e: symlinks: create recursive symlink \(should return error\) ========================= 18:51:22 \(1469497882\) [ 535.814703] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 18:51:22 (1469497882) [ 536.071358] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 536.527168] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 536.986736] Lustre: DEBUG MARKER: dmesg [ 537.452995] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17f: symlinks: long and very long symlink name ======================================== 18:51:24 \(1469497884\) [ 537.688826] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 18:51:24 (1469497884) [ 538.004179] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 538.448005] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 538.890432] Lustre: DEBUG MARKER: dmesg [ 539.309699] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 18:51:26 \(1469497886\) [ 539.474132] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 18:51:26 (1469497886) [ 539.631084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 539.910266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 540.180616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 540.758444] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 541.133586] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 541.560509] Lustre: DEBUG MARKER: dmesg [ 542.031123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17h: create objects: lov_free_memmd\(\) doesn\'t lbug ==================================== 18:51:28 \(1469497888\) [ 542.223654] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 18:51:28 (1469497888) [ 542.441382] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000141 [ 542.786626] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 543.165138] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 543.560134] Lustre: DEBUG MARKER: dmesg [ 544.080669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17i: don\'t panic on short symlink ===================================================== 18:51:30 \(1469497890\) [ 544.244115] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink ===================================================== 18:51:30 (1469497890) [ 544.496648] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000143 [ 544.659182] Lustre: *** cfs_fail_loc=143, val=0*** [ 544.885821] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 545.323952] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 545.780697] Lustre: DEBUG MARKER: dmesg [ 546.297042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 18:51:33 \(1469497893\) [ 546.500072] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 18:51:33 (1469497893) [ 546.980866] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 547.417769] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 547.828381] Lustre: DEBUG MARKER: dmesg [ 548.239488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17l: Ensure lgetxattr\'s returned xattr size is consistent ============================= 18:51:35 \(1469497895\) [ 548.491760] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ============================= 18:51:35 (1469497895) [ 548.775418] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 549.146753] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 549.560830] Lustre: DEBUG MARKER: dmesg [ 550.090717] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17m: run e2fsck against MDT which contains short\/long symlink ========================= 18:51:36 \(1469497896\) [ 550.331101] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 18:51:36 (1469497896) [ 550.534581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 550.849330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 551.221676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_17m only for ldiskfs MDT [ 551.428568] Lustre: DEBUG MARKER: SKIP: sanity test_17m only for ldiskfs MDT [ 551.679650] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 552.079410] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 552.498582] Lustre: DEBUG MARKER: dmesg [ 553.009900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17n: run e2fsck against master\/slave MDT which contains remote dir ==================== 18:51:39 \(1469497899\) [ 553.216135] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ==================== 18:51:39 (1469497899) [ 553.417166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 553.754659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 554.183022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_17n only for ldiskfs MDT [ 554.463633] Lustre: DEBUG MARKER: SKIP: sanity test_17n only for ldiskfs MDT [ 554.670804] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 555.062970] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 555.552629] Lustre: DEBUG MARKER: dmesg [ 556.045987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17o: stat file with incompat LMA feature ============================================== 18:51:42 \(1469497902\) [ 556.264225] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature ============================================== 18:51:42 (1469497902) [ 556.460106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 556.848135] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 557.166940] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 557.332939] Lustre: Failing over lustre-MDT0000 [ 566.054853] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.86@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.814092] Lustre: 31643:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469497910/real 1469497910] req@ffff8800534fc000 x1540879853576464/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469497916 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 570.820529] Lustre: 31643:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 571.054163] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.86@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.284602] Lustre: server umount lustre-MDT0000 complete [ 577.441164] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 577.758034] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 603.985354] Lustre: DEBUG MARKER: hostname [ 604.408464] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 609.889138] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 614.960365] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 615.196577] Lustre: MGS: Connection restored to f14df7fe-c07f-3b6c-2e32-0b0fa5cf5034 (at 0@lo) [ 615.202624] Lustre: Skipped 8 previous similar messages [ 617.560535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 617.870555] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 618.470383] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 618.778529] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 619.081177] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 619.389625] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 631.453760] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 635.693571] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 636.382740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 636.536483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 636.564474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 636.706122] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 637.990884] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x194 [ 638.138907] Lustre: *** cfs_fail_loc=194, val=0*** [ 638.142178] Lustre: 776:0:(osd_object.c:395:osd_check_lma()) lustre-MDT0000: unsupported incompat LMA feature(s) 0x0 for fid = [0x200000bd0:0xbe:0x0] [ 638.290311] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 638.659044] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 639.059428] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 639.463302] Lustre: DEBUG MARKER: dmesg [ 639.970748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 18: touch ...\/f \; ls ... ============================================================== 18:53:06 \(1469497986\) [ 640.144341] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 18:53:06 (1469497986) [ 640.403239] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 640.824770] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 641.252428] Lustre: DEBUG MARKER: dmesg [ 641.780285] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19a: touch ...\/f19 \; ls -l ... \; rm ...\/f19 =========================================== 18:53:08 \(1469497988\) [ 641.972068] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 18:53:08 (1469497988) [ 642.302162] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 642.698180] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 643.080648] Lustre: DEBUG MARKER: dmesg [ 643.567416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19b: ls -l ...\/f19 \(should return error\) ============================================== 18:53:10 \(1469497990\) [ 643.792845] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 18:53:10 (1469497990) [ 644.000717] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 644.359173] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 644.778878] Lustre: DEBUG MARKER: dmesg [ 645.222459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19c: runas -u 500 -g 500 touch ...\/f19 \(should return error\) ========================== 18:53:12 \(1469497992\) [ 645.399894] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ========================== 18:53:12 (1469497992) [ 645.614183] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 646.080193] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 646.534914] Lustre: DEBUG MARKER: dmesg [ 646.996410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19d: cat ...\/f19 \(should return error\) ================================================ 18:53:13 \(1469497993\) [ 647.206505] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 18:53:13 (1469497993) [ 647.423861] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 647.852972] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 648.260121] Lustre: DEBUG MARKER: dmesg [ 648.787342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 20: touch ...\/f \; ls -l ... =========================================================== 18:53:15 \(1469497995\) [ 648.967346] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 18:53:15 (1469497995) [ 649.161862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 1 done [ 649.423425] Lustre: DEBUG MARKER: 1 done [ 649.639884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 2 done [ 649.841443] Lustre: DEBUG MARKER: 2 done [ 650.076266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 3 done [ 650.289948] Lustre: DEBUG MARKER: 3 done [ 650.497093] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 650.873968] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 651.350978] Lustre: DEBUG MARKER: dmesg [ 651.860826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 21: write to dangling link ============================================================ 18:53:18 \(1469497998\) [ 652.063197] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 18:53:18 (1469497998) [ 652.392037] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 652.814576] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 653.289216] Lustre: DEBUG MARKER: dmesg [ 653.755783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 22: unpack tar archive as non-root user =============================================== 18:53:20 \(1469498000\) [ 653.940044] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 18:53:20 (1469498000) [ 654.732559] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 655.119126] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 655.534289] Lustre: DEBUG MARKER: dmesg [ 655.987151] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 23a: O_CREAT\|O_EXCL in subdir ========================================================= 18:53:22 \(1469498002\) [ 656.202462] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 18:53:22 (1469498002) [ 656.437910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 656.907146] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 657.349209] Lustre: DEBUG MARKER: dmesg [ 657.826738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 23b: O_APPEND check =================================================================== 18:53:24 \(1469498004\) [ 658.014568] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 18:53:24 (1469498004) [ 658.309589] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 658.748369] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 659.186605] Lustre: DEBUG MARKER: dmesg [ 659.697276] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24a: rename file to non-existent target =============================================== 18:53:26 \(1469498006\) [ 659.919640] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target =============================================== 18:53:26 (1469498006) [ 660.139140] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 660.559342] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 660.990652] Lustre: DEBUG MARKER: dmesg [ 661.438571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24b: rename file to existing target =================================================== 18:53:28 \(1469498008\) [ 661.613592] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target =================================================== 18:53:28 (1469498008) [ 661.900602] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 662.300095] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 662.720019] Lustre: DEBUG MARKER: dmesg [ 663.264577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24c: rename directory to non-existent target ========================================== 18:53:30 \(1469498010\) [ 663.482236] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================== 18:53:30 (1469498010) [ 663.798589] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 664.283417] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 664.709875] Lustre: DEBUG MARKER: dmesg [ 665.192943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24d: rename directory to existing target ============================================== 18:53:31 \(1469498011\) [ 665.401886] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target ============================================== 18:53:31 (1469498011) [ 665.668264] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 666.050696] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 666.433296] Lustre: DEBUG MARKER: dmesg [ 666.838361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24e: touch ...\/R5a\/f\; rename ...\/R5a\/f ...\/R5b\/g ====================================== 18:53:33 \(1469498013\) [ 667.072087] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 18:53:33 (1469498013) [ 667.380659] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 667.852153] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 668.361928] Lustre: DEBUG MARKER: dmesg [ 668.868176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24f: touch ...\/R6a\/f R6b\/g\; mv ...\/R6a\/f ...\/R6b\/g ==================================== 18:53:35 \(1469498015\) [ 669.068790] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 18:53:35 (1469498015) [ 669.355721] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 669.787394] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 670.169537] Lustre: DEBUG MARKER: dmesg [ 670.685709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24g: mkdir ...\/R7{a,b}\/d\; mv ...\/R7a\/d ...\/R7b\/e ====================================== 18:53:37 \(1469498017\) [ 670.891188] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 18:53:37 (1469498017) [ 671.189595] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 671.590636] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 671.958744] Lustre: DEBUG MARKER: dmesg [ 672.459591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24h: mkdir ...\/R8{a,b}\/{d,e}\; rename ...\/R8a\/d ...\/R8b\/e ============================== 18:53:39 \(1469498019\) [ 672.631189] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 18:53:39 (1469498019) [ 672.910800] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 673.273810] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 673.679715] Lustre: DEBUG MARKER: dmesg [ 674.179477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24i: rename file to dir error: touch f \; mkdir a \; rename f a ========================= 18:53:40 \(1469498020\) [ 674.355806] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 18:53:40 (1469498020) [ 674.659595] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 675.043996] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 675.431508] Lustre: DEBUG MARKER: dmesg [ 675.852201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24j: source does not exist ============================================================ 18:53:42 \(1469498022\) [ 676.035848] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 18:53:42 (1469498022) [ 676.322176] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 676.721613] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 677.178352] Lustre: DEBUG MARKER: dmesg [ 677.666068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24k: touch ...\/R11a\/f\; mv ...\/R11a\/f ...\/R11a\/d ======================================= 18:53:44 \(1469498024\) [ 677.835037] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 18:53:44 (1469498024) [ 678.118442] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 678.517678] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 678.938639] Lustre: DEBUG MARKER: dmesg [ 679.415947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24l: Renaming a file to itself ======================================================== 18:53:46 \(1469498026\) [ 679.653906] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 18:53:46 (1469498026) [ 679.912113] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 680.323204] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 680.714516] Lustre: DEBUG MARKER: dmesg [ 681.205968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24m: Renaming a file to a hard link to itself ========================================= 18:53:47 \(1469498027\) [ 681.392192] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 18:53:47 (1469498027) [ 681.672687] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 682.181907] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 682.596059] Lustre: DEBUG MARKER: dmesg [ 683.041229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24n: Statting the old file after renaming \(Posix rename 2\) ============================ 18:53:49 \(1469498029\) [ 683.216938] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 18:53:49 (1469498029) [ 683.536938] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 683.965082] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 684.411097] Lustre: DEBUG MARKER: dmesg [ 684.925713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24o: rename of files during htree split =============================================== 18:53:51 \(1469498031\) [ 685.112489] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 18:53:51 (1469498031) [ 787.821495] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 788.298165] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 788.758779] Lustre: DEBUG MARKER: dmesg [ 789.441736] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24p: mkdir ...\/R12{a,b}\; rename ...\/R12a ...\/R12b ===================================== 18:55:36 \(1469498136\) [ 789.648053] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 18:55:36 (1469498136) [ 789.956555] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 790.427724] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 790.850665] Lustre: DEBUG MARKER: dmesg [ 791.326863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24q: mkdir ...\/R13{a,b}\; open R13b rename R13a R13b =================================== 18:55:38 \(1469498138\) [ 791.516864] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 18:55:38 (1469498138) [ 791.837760] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 792.271507] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 792.720074] Lustre: DEBUG MARKER: dmesg [ 793.190382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24r: mkdir ...\/R14a\/b\; rename ...\/R14a ...\/R14a\/b ===================================== 18:55:39 \(1469498139\) [ 793.435887] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 18:55:39 (1469498139) [ 793.701945] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 794.110871] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 794.521324] Lustre: DEBUG MARKER: dmesg [ 794.993168] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24s: mkdir ...\/R15a\/b\/c\; rename ...\/R15a ...\/R15a\/b\/c ================================= 18:55:41 \(1469498141\) [ 795.176403] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 18:55:41 (1469498141) [ 795.477652] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 795.916222] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 796.361493] Lustre: DEBUG MARKER: dmesg [ 796.898901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24t: mkdir ...\/R16a\/b\/c\; rename ...\/R16a\/b\/c ...\/R16a ================================= 18:55:43 \(1469498143\) [ 797.107980] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 18:55:43 (1469498143) [ 797.367641] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 797.790314] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 798.271695] Lustre: DEBUG MARKER: dmesg [ 798.748660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24u: create stripe file =============================================================== 18:55:45 \(1469498145\) [ 798.939143] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 18:55:45 (1469498145) [ 799.208783] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 799.649288] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 800.036224] Lustre: DEBUG MARKER: dmesg [ 800.517149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24v: list directory with large files \(handle hash collision, bug: 17560\) ============== 18:55:47 \(1469498147\) [ 800.720524] Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) ============== 18:55:47 (1469498147) [ 843.104462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 844.460110] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 844.843796] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 845.238925] Lustre: DEBUG MARKER: dmesg [ 845.734562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24w: Reading a file larger than 4Gb =================================================== 18:56:32 \(1469498192\) [ 845.949926] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 18:56:32 (1469498192) [ 846.204322] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 846.590474] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 846.972212] Lustre: DEBUG MARKER: dmesg [ 847.395124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24x: cross MDT rename\/link ============================================================ 18:56:34 \(1469498194\) [ 847.611498] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ============================================================ 18:56:34 (1469498194) [ 847.836246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24x needs \>= 2 MDTs [ 848.007958] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [ 848.211057] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 848.588806] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 848.999810] Lustre: DEBUG MARKER: dmesg [ 849.439275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24y: rename\/link on the same dir should succeed ======================================= 18:56:36 \(1469498196\) [ 849.612847] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ======================================= 18:56:36 (1469498196) [ 849.841055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24y needs \>= 2 MDTs [ 850.058206] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [ 850.242917] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 850.639488] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 851.070685] Lustre: DEBUG MARKER: dmesg [ 851.561125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24A: readdir\(\) returns correct number of entries. ===================================== 18:56:38 \(1469498198\) [ 851.750969] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ===================================== 18:56:38 (1469498198) [ 868.714689] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 869.096590] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 869.489773] Lustre: DEBUG MARKER: dmesg [ 869.945071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24B: readdir for striped dir return correct number of entries ========================= 18:56:56 \(1469498216\) [ 870.171274] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================= 18:56:56 (1469498216) [ 870.403690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24B needs \>= 2 MDTs [ 870.558504] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [ 870.771024] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 871.169829] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 871.550452] Lustre: DEBUG MARKER: dmesg [ 872.084957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24C: check .. in striped dir ========================================================== 18:56:58 \(1469498218\) [ 872.251238] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ========================================================== 18:56:58 (1469498218) [ 872.507614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24C needs \>= 2 MDTs [ 872.716888] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [ 872.958525] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 873.312399] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 873.683995] Lustre: DEBUG MARKER: dmesg [ 874.171753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24D skipping SLOW test 24D [ 874.330700] Lustre: DEBUG MARKER: SKIP: sanity test_24D skipping SLOW test 24D [ 874.714161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24E: cross MDT rename\/link ============================================================ 18:57:01 \(1469498221\) [ 874.940214] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ============================================================ 18:57:01 (1469498221) [ 875.139752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24E needs \>= 4 MDTs [ 875.332558] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 875.539885] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 875.907119] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 876.326828] Lustre: DEBUG MARKER: dmesg [ 876.750762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 25a: create file in symlinked directory =============================================== 18:57:03 \(1469498223\) [ 876.907035] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 18:57:03 (1469498223) [ 877.143982] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 877.493639] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 877.904126] Lustre: DEBUG MARKER: dmesg [ 878.396092] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 25b: lookup file in symlinked directory =============================================== 18:57:05 \(1469498225\) [ 878.559845] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 18:57:05 (1469498225) [ 878.768431] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 879.197529] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 879.594377] Lustre: DEBUG MARKER: dmesg [ 880.059717] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26a: multiple component symlink ======================================================= 18:57:06 \(1469498226\) [ 880.246768] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 18:57:06 (1469498226) [ 880.510333] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 880.880292] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 881.325430] Lustre: DEBUG MARKER: dmesg [ 881.840911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26b: multiple component symlink at end of lookup ====================================== 18:57:08 \(1469498228\) [ 882.047307] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 18:57:08 (1469498228) [ 882.259958] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 882.636903] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 883.044372] Lustre: DEBUG MARKER: dmesg [ 883.522582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26c: chain of symlinks ================================================================ 18:57:10 \(1469498230\) [ 883.735313] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 18:57:10 (1469498230) [ 884.009965] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 884.389739] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 884.778789] Lustre: DEBUG MARKER: dmesg [ 885.275732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26d: create multiple component recursive symlink ====================================== 18:57:12 \(1469498232\) [ 885.454373] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 18:57:12 (1469498232) [ 885.637200] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 886.009137] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 886.366807] Lustre: DEBUG MARKER: dmesg [ 886.761723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26e: unlink multiple component recursive symlink ====================================== 18:57:13 \(1469498233\) [ 886.942149] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 18:57:13 (1469498233) [ 887.141087] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 887.494874] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 887.859967] Lustre: DEBUG MARKER: dmesg [ 888.251107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 18:57:15 \(1469498235\) [ 888.446291] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 18:57:15 (1469498235) [ 888.712441] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 889.107826] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 889.502023] Lustre: DEBUG MARKER: dmesg [ 889.913993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27a: one stripe file ================================================================== 18:57:16 \(1469498236\) [ 890.141391] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 18:57:16 (1469498236) [ 890.381121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == test_27a: write to one stripe file ========================= [ 890.579483] Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= [ 890.794535] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 891.189497] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 891.564148] Lustre: DEBUG MARKER: dmesg [ 891.959128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27b: create and write to two stripe file ============================================== 18:57:18 \(1469498238\) [ 892.128933] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file ============================================== 18:57:18 (1469498238) [ 892.350043] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 892.725147] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 893.097028] Lustre: DEBUG MARKER: dmesg [ 893.512732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27d: create file with default settings ================================================ 18:57:20 \(1469498240\) [ 893.685570] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 18:57:20 (1469498240) [ 893.927913] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 894.315491] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 894.739774] Lustre: DEBUG MARKER: dmesg [ 895.174110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27e: setstripe existing file \(should return error\) ==================================== 18:57:22 \(1469498242\) [ 895.338167] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 18:57:22 (1469498242) [ 895.479445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 895.819462] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 896.200630] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 896.547791] Lustre: DEBUG MARKER: dmesg [ 896.939334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27f: setstripe with bad stripe size \(should return error\) ============================= 18:57:23 \(1469498243\) [ 897.106830] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 18:57:23 (1469498243) [ 897.312996] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 897.660672] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 898.018629] Lustre: DEBUG MARKER: dmesg [ 898.427731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27g: \/usr\/bin\/lfs getstripe with no objects =========================================== 18:57:25 \(1469498245\) [ 898.597202] Lustre: DEBUG MARKER: == sanity test 27g: /usr/bin/lfs getstripe with no objects =========================================== 18:57:25 (1469498245) [ 898.803976] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 899.162901] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 899.530937] Lustre: DEBUG MARKER: dmesg [ 899.963422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27i: \/usr\/bin\/lfs getstripe with some objects ========================================= 18:57:26 \(1469498246\) [ 900.141616] Lustre: DEBUG MARKER: == sanity test 27i: /usr/bin/lfs getstripe with some objects ========================================= 18:57:26 (1469498246) [ 900.337934] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 900.706872] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 901.089061] Lustre: DEBUG MARKER: dmesg [ 901.495678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27j: setstripe with bad stripe offset \(should return error\) =========================== 18:57:28 \(1469498248\) [ 901.660885] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 18:57:28 (1469498248) [ 901.677939] LustreError: 775:0:(lod_qos.c:1227:lod_alloc_specific()) Start index 2 not found in pool '' [ 901.895804] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 902.268812] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 902.685281] Lustre: DEBUG MARKER: dmesg [ 903.169700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27k: limit i_blksize for broken user apps ============================================= 18:57:30 \(1469498250\) [ 903.341388] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 18:57:30 (1469498250) [ 903.547875] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 903.958490] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 904.357892] Lustre: DEBUG MARKER: dmesg [ 904.759376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27l: check setstripe permissions \(should return error\) ================================ 18:57:31 \(1469498251\) [ 904.937645] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 18:57:31 (1469498251) [ 905.148607] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 905.521666] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 905.943082] Lustre: DEBUG MARKER: dmesg [ 906.471138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_27m skipping SLOW test 27m [ 906.660156] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 906.885831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27n: create file with some full OSTs ================================================== 18:57:33 \(1469498253\) [ 907.145538] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 18:57:33 (1469498253) [ 908.131478] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 918.434212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 918.706880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 918.977698] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 919.246328] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 919.516096] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 919.792737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 920.067258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 920.341814] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 920.611146] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 921.275781] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 921.847846] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 932.671365] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 943.031034] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 943.413236] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 943.882660] Lustre: DEBUG MARKER: dmesg [ 944.335868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27o: create file with all full OSTs \(should error\) ==================================== 18:58:11 \(1469498291\) [ 944.533922] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 18:58:11 (1469498291) [ 945.558501] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 955.905955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 956.203574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 956.519758] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 956.840434] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 957.141089] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 957.429641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 957.741212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 958.052181] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 958.357857] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 959.122554] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 959.770577] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 970.112639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 970.420915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 970.730971] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 971.047171] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 971.342531] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 971.660765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 971.979898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 972.300326] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 972.627960] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 973.510413] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 974.150507] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 985.110837] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 995.766639] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 996.169286] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 996.636067] Lustre: DEBUG MARKER: dmesg [ 997.141174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27p: append to a truncated file with some full OSTs =================================== 18:59:03 \(1469498343\) [ 997.329213] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 18:59:03 (1469498343) [ 998.452842] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1008.867744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1009.180838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1009.489982] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1009.789932] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1010.110558] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1010.420056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1010.739982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1011.070845] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1011.358753] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1012.210890] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1012.864730] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1023.850931] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1034.230911] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1034.721400] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1035.130658] Lustre: DEBUG MARKER: dmesg [ 1035.635551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27q: append to truncated file with all OSTs full \(should error\) ======================= 18:59:42 \(1469498382\) [ 1035.862748] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 18:59:42 (1469498382) [ 1036.883060] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1047.310209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1047.609679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1047.938224] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1048.259873] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1048.570611] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1048.949030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1049.261570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1049.590112] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1049.891522] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1050.719845] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1051.400849] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1061.739366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1062.102157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1062.451991] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1062.790145] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1063.092370] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 1063.411396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1063.711978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1064.027706] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1064.342149] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1065.083681] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1065.710406] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1076.652582] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1087.087393] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1087.468346] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1087.918893] Lustre: DEBUG MARKER: dmesg [ 1088.379884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27r: stripe file with some full OSTs \(shouldn\'t LBUG\) ================================= 19:00:35 \(1469498435\) [ 1088.550249] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 19:00:35 (1469498435) [ 1089.609084] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1099.949789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1100.300807] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1100.663277] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1100.961324] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1101.271863] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1101.579978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1101.910257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1102.229829] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1102.520937] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1103.331546] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1104.009798] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1114.993529] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1125.347081] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1125.746612] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1126.183136] Lustre: DEBUG MARKER: dmesg [ 1127.592244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27s: lsm_xfersize overflow \(should error\) \(bug 10725\) ================================= 19:01:14 \(1469498474\) [ 1127.772862] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 19:01:14 (1469498474) [ 1127.967580] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1128.312959] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1128.721934] Lustre: DEBUG MARKER: dmesg [ 1129.147294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27t: check that utils parse path correctly ============================================ 19:01:15 \(1469498475\) [ 1129.320305] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 19:01:15 (1469498475) [ 1129.520637] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1129.899311] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1130.315468] Lustre: DEBUG MARKER: dmesg [ 1130.760842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27u: skip object creation on OSC w\/o objects ========================================== 19:01:17 \(1469498477\) [ 1130.952653] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================== 19:01:17 (1469498477) [ 1131.239876] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x139 [ 1131.422889] Lustre: *** cfs_fail_loc=139, val=0*** [ 1131.924082] Lustre: *** cfs_fail_loc=139, val=0*** [ 1131.926334] Lustre: Skipped 274 previous similar messages [ 1132.927318] Lustre: *** cfs_fail_loc=139, val=0*** [ 1132.931973] Lustre: Skipped 559 previous similar messages [ 1134.921027] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 1145.417939] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1145.910892] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1146.310560] Lustre: DEBUG MARKER: dmesg [ 1146.811055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27v: skip object creation on slow OST ================================================= 19:01:33 \(1469498493\) [ 1147.042596] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 19:01:33 (1469498493) [ 1147.278092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1147.569975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1147.890576] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1148.189313] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1148.512482] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1148.818967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1149.150877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1149.450373] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1149.739294] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1150.501570] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1151.162777] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1161.520424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1161.849899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1162.140184] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1162.437489] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1162.771340] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 1163.092761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1163.441502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1163.760795] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1164.081434] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1164.859913] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1165.500643] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1176.600838] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1197.777331] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1208.102554] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1208.481345] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1208.906671] Lustre: DEBUG MARKER: dmesg [ 1209.322709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27w: check \/usr\/bin\/lfs setstripe -S option =========================================== 19:02:36 \(1469498556\) [ 1209.496106] Lustre: DEBUG MARKER: == sanity test 27w: check /usr/bin/lfs setstripe -S option =========================================== 19:02:36 (1469498556) [ 1209.734293] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1210.086817] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1210.439605] Lustre: DEBUG MARKER: dmesg [ 1210.823040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27wa: check \/usr\/bin\/lfs setstripe -c -i options ====================================== 19:02:37 \(1469498557\) [ 1211.033627] Lustre: DEBUG MARKER: == sanity test 27wa: check /usr/bin/lfs setstripe -c -i options ====================================== 19:02:37 (1469498557) [ 1211.281075] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1211.730925] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1212.139363] Lustre: DEBUG MARKER: dmesg [ 1212.574937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27x: create files while OST0 is degraded ============================================== 19:02:39 \(1469498559\) [ 1212.761745] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 19:02:39 (1469498559) [ 1213.330690] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1223.995828] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1224.420075] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1224.817083] Lustre: DEBUG MARKER: dmesg [ 1225.297575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 19:02:52 \(1469498572\) [ 1225.511773] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 19:02:52 (1469498572) [ 1225.727724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1226.032509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1226.368129] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1226.658785] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1226.958430] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1227.233592] Lustre: DEBUG MARKER: lctl dl [ 1227.533310] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 deactivate [ 1227.664499] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1228.174099] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1238.776813] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 1238.909237] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.2.5.85@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1238.920489] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1238.924405] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 10.2.5.85@tcp (at 10.2.5.85@tcp) [ 1238.926735] Lustre: Skipped 8 previous similar messages [ 1239.061361] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1249.479913] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1249.909000] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1250.387951] Lustre: DEBUG MARKER: dmesg [ 1250.843602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27z: check SEQ\/OID on the MDT and OST filesystems ===================================== 19:03:17 \(1469498597\) [ 1251.082769] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 19:03:17 (1469498597) [ 1257.521038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check file \/mnt\/lustre\/d27z.sanity\/f27z.sanity-1 [ 1257.701696] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 1258.017482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark FID seq 0x200000bd0, oid 0x6850 ver 0x0 [ 1258.195394] Lustre: DEBUG MARKER: FID seq 0x200000bd0, oid 0x6850 ver 0x0 [ 1258.460234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LOV seq 0x200000bd0, oid 0x6850, count: 1 [ 1258.673842] Lustre: DEBUG MARKER: LOV seq 0x200000bd0, oid 0x6850, count: 1 [ 1258.907523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:0 ost:0 oid:602\/0x25a seq:0 [ 1259.100365] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:602/0x25a seq:0 [ 1261.776522] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.2.5.85@tcp failed: rc = -107 [ 1261.783048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.2.5.85@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1278.450903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check file \/mnt\/lustre\/d27z.sanity\/f27z.sanity-2 [ 1278.643879] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 1278.929533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark FID seq 0x200000bd0, oid 0x6852 ver 0x0 [ 1279.118669] Lustre: DEBUG MARKER: FID seq 0x200000bd0, oid 0x6852 ver 0x0 [ 1279.378281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LOV seq 0x200000bd0, oid 0x6852, count: 2 [ 1279.581831] Lustre: DEBUG MARKER: LOV seq 0x200000bd0, oid 0x6852, count: 2 [ 1279.819520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:0 ost:1 oid:1539\/0x603 seq:0 [ 1279.991966] Lustre: DEBUG MARKER: want: stripe:0 ost:1 oid:1539/0x603 seq:0 [ 1284.000603] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 10.2.5.85@tcp failed: rc = -107 [ 1284.003649] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.2.5.85@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1296.729706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:1 ost:0 oid:603\/0x25b seq:0 [ 1296.950871] Lustre: DEBUG MARKER: want: stripe:1 ost:0 oid:603/0x25b seq:0 [ 1302.351944] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.2.5.85@tcp failed: rc = -107 [ 1302.354549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.2.5.85@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1315.664706] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1316.061679] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1316.529371] Lustre: DEBUG MARKER: dmesg [ 1316.946728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 19:04:23 \(1469498663\) [ 1317.136706] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 19:04:23 (1469498663) [ 1317.188415] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1317.190970] Lustre: Skipped 4 previous similar messages [ 1317.408387] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1317.825398] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1318.201362] Lustre: DEBUG MARKER: dmesg [ 1318.650694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27B: call setstripe on open unlinked file\/rename victim =============================== 19:04:25 \(1469498665\) [ 1318.844405] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim =============================== 19:04:25 (1469498665) [ 1319.069299] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1319.286534] LustreError: 1760:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 1319.456622] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1319.808692] Lustre: DEBUG MARKER: dmesg [ 1320.221780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27C: check full striping across all OSTs ============================================== 19:04:27 \(1469498667\) [ 1320.411509] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs ============================================== 19:04:27 (1469498667) [ 1320.650722] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1321.072159] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1321.421036] Lustre: DEBUG MARKER: dmesg [ 1321.859523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27D: validate llapi_layout API ======================================================== 19:04:28 \(1469498668\) [ 1322.045392] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API ======================================================== 19:04:28 (1469498668) [ 1322.207653] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1328.536845] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.testpool lustre-OST[0-1/1] [ 1339.604700] Lustre: DEBUG MARKER: lctl pool_list lustre [ 1339.882849] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool [ 1340.154751] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool [ 1340.426893] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1344.720952] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [ 1353.031525] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1361.359580] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1361.759054] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1362.182131] Lustre: DEBUG MARKER: dmesg [ 1362.630530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27E: check that default extended attribute size properly increases ==================== 19:05:09 \(1469498709\) [ 1362.806261] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ==================== 19:05:09 (1469498709) [ 1363.037734] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1363.415708] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1363.779835] Lustre: DEBUG MARKER: dmesg [ 1364.203563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 19:05:11 \(1469498711\) [ 1364.381914] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 19:05:11 (1469498711) [ 1364.544238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1374.893066] Lustre: 7955:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469498713/real 1469498713] req@ffff880049ac7300 x1540879853612384/t0(0) o13->lustre-OST0001-osc-MDT0000@10.2.5.85@tcp:7/4 lens 224/368 e 0 to 1 dl 1469498720 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 1374.898969] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.2.5.85@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1382.646153] Lustre: 2278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469498716/real 1469498716] req@ffff88004f989e00 x1540879853612464/t0(0) o10->lustre-OST0000-osc-MDT0000@10.2.5.85@tcp:6/4 lens 560/432 e 0 to 1 dl 1469498728 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1382.654597] Lustre: 2278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1397.401584] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1397.853321] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1398.349962] Lustre: DEBUG MARKER: dmesg [ 1398.839902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 28: create\/mknod\/mkdir with bad file types ============================================ 19:05:45 \(1469498745\) [ 1399.027838] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 19:05:45 (1469498745) [ 1399.520062] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1399.921645] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1400.308167] Lustre: DEBUG MARKER: dmesg [ 1400.789343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 29: IT_GETATTR regression ============================================================ 19:05:47 \(1469498747\) [ 1400.980289] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 19:05:47 (1469498747) [ 1402.709614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark first d29 [ 1402.887534] Lustre: DEBUG MARKER: first d29 [ 1403.136965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark second d29 [ 1403.325088] Lustre: DEBUG MARKER: second d29 [ 1403.549471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done [ 1403.732826] Lustre: DEBUG MARKER: done [ 1403.928495] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1404.339225] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1404.740912] Lustre: DEBUG MARKER: dmesg [ 1405.193335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30a: execute binary from Lustre \(execve\) ============================================== 19:05:51 \(1469498751\) [ 1405.382381] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 19:05:51 (1469498751) [ 1405.629521] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1406.009783] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1406.420954] Lustre: DEBUG MARKER: dmesg [ 1406.931239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30b: execute binary from Lustre as non-root =========================================== 19:05:53 \(1469498753\) [ 1407.100895] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 19:05:53 (1469498753) [ 1407.391052] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1407.771858] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1408.191364] Lustre: DEBUG MARKER: dmesg [ 1408.647974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30c: execute binary from Lustre without read perms ==================================== 19:05:55 \(1469498755\) [ 1408.830529] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 19:05:55 (1469498755) [ 1409.121446] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1409.574913] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1410.029985] Lustre: DEBUG MARKER: dmesg [ 1410.489653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31a: open-unlink file ================================================================= 19:05:57 \(1469498757\) [ 1410.690209] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 19:05:57 (1469498757) [ 1410.948087] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1411.379779] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1411.836945] Lustre: DEBUG MARKER: dmesg [ 1412.350204] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31b: unlink file with multiple links while open ======================================= 19:05:59 \(1469498759\) [ 1412.545340] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 19:05:59 (1469498759) [ 1412.840582] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1413.268849] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1413.685933] Lustre: DEBUG MARKER: dmesg [ 1414.172528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31c: open-unlink file with multiple links ============================================= 19:06:00 \(1469498760\) [ 1414.363428] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 19:06:00 (1469498760) [ 1414.648464] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1415.050827] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1415.472792] Lustre: DEBUG MARKER: dmesg [ 1415.872224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31d: remove of open directory ========================================================= 19:06:02 \(1469498762\) [ 1416.072024] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 19:06:02 (1469498762) [ 1416.394593] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1416.821704] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1417.225218] Lustre: DEBUG MARKER: dmesg [ 1417.673844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31e: remove of open non-empty directory =============================================== 19:06:04 \(1469498764\) [ 1417.857041] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 19:06:04 (1469498764) [ 1418.086790] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1418.477465] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1418.892578] Lustre: DEBUG MARKER: dmesg [ 1419.394079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31f: remove of open directory with open-unlink file =================================== 19:06:06 \(1469498766\) [ 1419.569265] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 19:06:06 (1469498766) [ 1425.959994] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1426.319272] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1426.682021] Lustre: DEBUG MARKER: dmesg [ 1427.091812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31g: cross directory link============================================================== 19:06:13 \(1469498773\) [ 1427.292675] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 19:06:13 (1469498773) [ 1427.522338] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1427.894919] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1428.254559] Lustre: DEBUG MARKER: dmesg [ 1428.815462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31h: cross directory link under child================================================== 19:06:15 \(1469498775\) [ 1428.981992] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 19:06:15 (1469498775) [ 1429.178876] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1429.535444] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1429.897966] Lustre: DEBUG MARKER: dmesg [ 1430.289854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31i: cross directory link under parent================================================= 19:06:17 \(1469498777\) [ 1430.483039] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 19:06:17 (1469498777) [ 1430.711755] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1431.122621] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1431.574525] Lustre: DEBUG MARKER: dmesg [ 1432.027808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31j: link for directory================================================================ 19:06:18 \(1469498778\) [ 1432.227277] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 19:06:18 (1469498778) [ 1432.483722] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1432.875575] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1433.361829] Lustre: DEBUG MARKER: dmesg [ 1433.831053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31k: link to file: the same, non-existing, dir========================================= 19:06:20 \(1469498780\) [ 1434.022922] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 19:06:20 (1469498780) [ 1434.352760] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1434.733078] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1435.133116] Lustre: DEBUG MARKER: dmesg [ 1435.708888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31m: link to file: the same, non-existing, dir========================================= 19:06:22 \(1469498782\) [ 1435.882137] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 19:06:22 (1469498782) [ 1436.201714] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1436.597835] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1437.005243] Lustre: DEBUG MARKER: dmesg [ 1437.522773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31n: check link count of unlinked file ================================================ 19:06:24 \(1469498784\) [ 1437.713298] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ================================================ 19:06:24 (1469498784) [ 1437.929761] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1438.322461] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1438.712701] Lustre: DEBUG MARKER: dmesg [ 1439.200628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31o: duplicate hard links with same filename ========================================== 19:06:25 \(1469498785\) [ 1439.402098] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================== 19:06:25 (1469498785) [ 1448.420660] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1448.913278] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1449.305777] Lustre: DEBUG MARKER: dmesg [ 1449.730743] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31p: remove of open striped directory ================================================= 19:06:36 \(1469498796\) [ 1449.931666] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ================================================= 19:06:36 (1469498796) [ 1450.135482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_31p needs \>= 2 MDTs [ 1450.338273] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [ 1450.548690] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1450.911070] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1451.268151] Lustre: DEBUG MARKER: dmesg [ 1451.770105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32a: stat d32a\/ext2-mountpoint\/.. ===================================================== 19:06:38 \(1469498798\) [ 1451.956254] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 19:06:38 (1469498798) [ 1452.500811] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1452.925431] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1453.335882] Lustre: DEBUG MARKER: dmesg [ 1453.865581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32b: open d32b\/ext2-mountpoint\/.. ===================================================== 19:06:40 \(1469498800\) [ 1454.072082] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 19:06:40 (1469498800) [ 1454.497684] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1454.950574] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1455.360929] Lustre: DEBUG MARKER: dmesg [ 1455.885370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32c: stat d32c\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 19:06:42 \(1469498802\) [ 1456.115232] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 19:06:42 (1469498802) [ 1456.637149] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1457.080952] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1457.537088] Lustre: DEBUG MARKER: dmesg [ 1457.995646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32d: open d32d\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 19:06:44 \(1469498804\) [ 1458.184360] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 19:06:44 (1469498804) [ 1458.680059] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1459.102808] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1459.469605] Lustre: DEBUG MARKER: dmesg [ 1459.956386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32e: stat d32e\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 19:06:46 \(1469498806\) [ 1460.151533] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 19:06:46 (1469498806) [ 1460.456024] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1460.848149] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1461.263141] Lustre: DEBUG MARKER: dmesg [ 1461.754435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32f: open d32f\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 19:06:48 \(1469498808\) [ 1461.938993] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 19:06:48 (1469498808) [ 1462.247659] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1462.678834] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1463.150056] Lustre: DEBUG MARKER: dmesg [ 1463.613224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32g: stat d32g\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 19:06:50 \(1469498810\) [ 1463.820509] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 19:06:50 (1469498810) [ 1464.126691] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1464.603479] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1465.081207] Lustre: DEBUG MARKER: dmesg [ 1465.601763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32h: open d32h\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 19:06:52 \(1469498812\) [ 1465.793132] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 19:06:52 (1469498812) [ 1466.084603] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1466.524765] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1466.941424] Lustre: DEBUG MARKER: dmesg [ 1467.417487] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32i: stat d32i\/ext2-mountpoint\/..\/test_file =========================================== 19:06:54 \(1469498814\) [ 1467.617553] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 19:06:54 (1469498814) [ 1468.072237] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1468.510682] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1468.998792] Lustre: DEBUG MARKER: dmesg [ 1469.483700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32j: open d32j\/ext2-mountpoint\/..\/test_file =========================================== 19:06:56 \(1469498816\) [ 1469.723535] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 19:06:56 (1469498816) [ 1470.324974] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1470.750687] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1471.202875] Lustre: DEBUG MARKER: dmesg [ 1471.691684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32k: stat d32k\/ext2-mountpoint\/..\/d2\/test_file ======================================== 19:06:58 \(1469498818\) [ 1471.902875] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 19:06:58 (1469498818) [ 1472.619145] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1473.009338] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1473.378185] Lustre: DEBUG MARKER: dmesg [ 1473.919314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32l: open d32l\/ext2-mountpoint\/..\/d2\/test_file ======================================== 19:07:00 \(1469498820\) [ 1474.105694] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 19:07:00 (1469498820) [ 1474.620893] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1475.072291] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1475.516801] Lustre: DEBUG MARKER: dmesg [ 1476.055526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32m: stat d32m\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 19:07:02 \(1469498822\) [ 1476.265952] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 19:07:02 (1469498822) [ 1476.560433] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1476.988738] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1477.370569] Lustre: DEBUG MARKER: dmesg [ 1477.831107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32n: open d32n\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 19:07:04 \(1469498824\) [ 1478.032124] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 19:07:04 (1469498824) [ 1478.332992] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1478.750455] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1479.188826] Lustre: DEBUG MARKER: dmesg [ 1479.710232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32o: stat d32o\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 19:07:06 \(1469498826\) [ 1479.933053] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 19:07:06 (1469498826) [ 1480.208701] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1480.610634] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1481.050873] Lustre: DEBUG MARKER: dmesg [ 1482.118595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32p: open d32p\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 19:07:08 \(1469498828\) [ 1482.317650] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 19:07:08 (1469498828) [ 1482.521548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_1 [ 1482.721063] Lustre: DEBUG MARKER: 32p_1 [ 1482.972967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_2 [ 1483.166404] Lustre: DEBUG MARKER: 32p_2 [ 1483.426205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_3 [ 1483.639946] Lustre: DEBUG MARKER: 32p_3 [ 1483.902549] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_4 [ 1484.081724] Lustre: DEBUG MARKER: 32p_4 [ 1484.407020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_5 [ 1484.605960] Lustre: DEBUG MARKER: 32p_5 [ 1484.832416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_6 [ 1485.010119] Lustre: DEBUG MARKER: 32p_6 [ 1485.291377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_7 [ 1485.469596] Lustre: DEBUG MARKER: 32p_7 [ 1485.705543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_8 [ 1485.882043] Lustre: DEBUG MARKER: 32p_8 [ 1486.122533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_9 [ 1486.316187] Lustre: DEBUG MARKER: 32p_9 [ 1486.539796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_10 [ 1486.718783] Lustre: DEBUG MARKER: 32p_10 [ 1486.985509] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1487.467804] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1487.866887] Lustre: DEBUG MARKER: dmesg [ 1488.348756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32q: stat follows mountpoints in Lustre \(should return error\) ========================= 19:07:15 \(1469498835\) [ 1488.574509] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 19:07:15 (1469498835) [ 1489.207599] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1489.674213] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1490.141053] Lustre: DEBUG MARKER: dmesg [ 1490.658816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32r: opendir follows mountpoints in Lustre \(should return error\) ====================== 19:07:17 \(1469498837\) [ 1490.876826] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 19:07:17 (1469498837) [ 1491.330815] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1491.766668] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1492.183419] Lustre: DEBUG MARKER: dmesg [ 1492.709142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33aa: write file with mode 444 \(should return error\) ================================== 19:07:19 \(1469498839\) [ 1492.896387] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ================================== 19:07:19 (1469498839) [ 1493.183970] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 33_1 [ 1493.370032] Lustre: DEBUG MARKER: 33_1 [ 1493.613188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 33_2 [ 1493.794414] Lustre: DEBUG MARKER: 33_2 [ 1494.008215] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1494.417739] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1494.826752] Lustre: DEBUG MARKER: dmesg [ 1495.374913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33a: test open file\(mode=0444\) with O_RDWR \(should return error\) ====================== 19:07:22 \(1469498842\) [ 1495.604801] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 19:07:22 (1469498842) [ 1507.241717] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1507.721165] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1508.124352] Lustre: DEBUG MARKER: dmesg [ 1508.579526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33b: test open file with malformed flags \(No panic\) =================================== 19:07:35 \(1469498855\) [ 1508.783557] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) =================================== 19:07:35 (1469498855) [ 1521.476490] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1521.920973] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1522.393766] Lustre: DEBUG MARKER: dmesg [ 1522.900768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33c: test llobdstat and write_bytes =================================================== 19:07:49 \(1469498869\) [ 1523.072071] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 19:07:49 (1469498869) [ 1524.009641] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1524.401484] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1524.794585] Lustre: DEBUG MARKER: dmesg [ 1525.292667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 19:07:52 \(1469498872\) [ 1525.491756] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 19:07:52 (1469498872) [ 1525.761696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33d needs \>= 2 MDTs [ 1525.952823] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [ 1526.176240] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1526.612070] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1526.998205] Lustre: DEBUG MARKER: dmesg [ 1527.497451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33e: mkdir and striped directory should have same mode ================================ 19:07:54 \(1469498874\) [ 1527.672061] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ================================ 19:07:54 (1469498874) [ 1527.912531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33e needs \>= 2 MDTs [ 1528.129851] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [ 1528.351514] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1528.776952] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1529.209198] Lustre: DEBUG MARKER: dmesg [ 1529.650941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 19:07:56 \(1469498876\) [ 1529.834893] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 19:07:56 (1469498876) [ 1530.110566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33f needs \>= 2 MDTs [ 1530.323960] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [ 1530.534002] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1530.969912] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1531.362929] Lustre: DEBUG MARKER: dmesg [ 1531.880769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34a: truncate file that has not been opened =========================================== 19:07:58 \(1469498878\) [ 1532.051180] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 19:07:58 (1469498878) [ 1532.280065] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1532.718370] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1533.089990] Lustre: DEBUG MARKER: dmesg [ 1533.516550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34b: O_RDONLY opening file doesn\'t create objects ===================================== 19:08:00 \(1469498880\) [ 1533.700383] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 19:08:00 (1469498880) [ 1533.929056] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1534.347152] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1534.844537] Lustre: DEBUG MARKER: dmesg [ 1535.351263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34c: O_RDWR opening file-with-size works ============================================== 19:08:02 \(1469498882\) [ 1535.551290] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 19:08:02 (1469498882) [ 1535.801898] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1536.208046] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1536.672737] Lustre: DEBUG MARKER: dmesg [ 1537.205366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34d: write to sparse file ============================================================= 19:08:03 \(1469498883\) [ 1537.389560] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 19:08:03 (1469498883) [ 1537.636935] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1538.039614] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1538.510563] Lustre: DEBUG MARKER: dmesg [ 1539.018472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34e: create objects, some with size and some without ================================== 19:08:05 \(1469498885\) [ 1539.213879] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 19:08:05 (1469498885) [ 1539.469375] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1539.917038] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1540.327532] Lustre: DEBUG MARKER: dmesg [ 1540.800652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34f: read from a file with no objects until EOF ======================================= 19:08:07 \(1469498887\) [ 1540.979701] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 19:08:07 (1469498887) [ 1541.264348] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1541.682666] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1542.140588] Lustre: DEBUG MARKER: dmesg [ 1542.680903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34g: truncate long file =============================================================== 19:08:09 \(1469498889\) [ 1542.874596] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 19:08:09 (1469498889) [ 1543.187113] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1543.579150] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1544.033958] Lustre: DEBUG MARKER: dmesg [ 1544.588833] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34h: ftruncate file under grouplock should not block ================================== 19:08:11 \(1469498891\) [ 1544.794567] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ================================== 19:08:11 (1469498891) [ 1548.317835] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1548.720414] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1549.107983] Lustre: DEBUG MARKER: dmesg [ 1549.640777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 35a: exec file with mode 444 \(should return and not leak\) ============================= 19:08:16 \(1469498896\) [ 1549.818376] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 19:08:16 (1469498896) [ 1550.125605] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1550.552868] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1550.995601] Lustre: DEBUG MARKER: dmesg [ 1551.499660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36a: MDS utime check \(mknod, utime\) =================================================== 19:08:18 \(1469498898\) [ 1551.694094] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 19:08:18 (1469498898) [ 1552.010980] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1552.461749] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1552.868483] Lustre: DEBUG MARKER: dmesg [ 1553.376934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36b: OST utime check \(open, utime\) ==================================================== 19:08:20 \(1469498900\) [ 1553.616253] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 19:08:20 (1469498900) [ 1553.880744] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1554.284567] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1554.702835] Lustre: DEBUG MARKER: dmesg [ 1555.153945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36c: non-root MDS utime check \(mknod, utime\) ========================================== 19:08:21 \(1469498901\) [ 1555.364468] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 19:08:21 (1469498901) [ 1568.115593] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1568.610025] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1569.000850] Lustre: DEBUG MARKER: dmesg [ 1569.535841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36d: non-root OST utime check \(open, utime\) =========================================== 19:08:36 \(1469498916\) [ 1569.735858] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 19:08:36 (1469498916) [ 1570.020848] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1570.490086] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1570.882055] Lustre: DEBUG MARKER: dmesg [ 1571.438746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36e: utime on non-owned file \(should return error\) ==================================== 19:08:38 \(1469498918\) [ 1571.629917] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 19:08:38 (1469498918) [ 1571.937993] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1572.362954] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1572.829818] Lustre: DEBUG MARKER: dmesg [ 1573.379995] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36f: utime on file racing with OST BRW write ========================================== 19:08:40 \(1469498920\) [ 1573.606971] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 19:08:40 (1469498920) [ 1574.961597] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1575.425637] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1575.897149] Lustre: DEBUG MARKER: dmesg [ 1576.438688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36g: filter mod data cache expiry ===================================================== 19:08:43 \(1469498923\) [ 1576.641214] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 19:08:43 (1469498923) [ 1619.815271] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1620.173487] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1620.542114] Lustre: DEBUG MARKER: dmesg [ 1620.944831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36h: utime on file racing with OST BRW write ========================================== 19:09:27 \(1469498967\) [ 1621.146356] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 19:09:27 (1469498967) [ 1622.474948] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1622.897203] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1623.346765] Lustre: DEBUG MARKER: dmesg [ 1623.916910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36i: change mtime on striped directory ================================================ 19:09:30 \(1469498970\) [ 1624.110934] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ================================================ 19:09:30 (1469498970) [ 1624.350617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_36i needs \>= 2 MDTs [ 1624.504411] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [ 1624.741679] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1625.199571] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1625.659849] Lustre: DEBUG MARKER: dmesg [ 1626.179509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 19:09:32 \(1469498972\) [ 1626.370537] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 19:09:32 (1469498972) [ 1626.596679] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1627.023521] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1627.431577] Lustre: DEBUG MARKER: dmesg [ 1627.954049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39: mtime changed on create =========================================================== 19:09:34 \(1469498974\) [ 1628.134589] Lustre: DEBUG MARKER: == sanity test 39: mtime changed on create =========================================================== 19:09:34 (1469498974) [ 1630.421150] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1630.857700] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1631.325311] Lustre: DEBUG MARKER: dmesg [ 1631.849385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 19:09:38 \(1469498978\) [ 1632.050741] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 19:09:38 (1469498978) [ 1633.631182] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1634.065841] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1634.524630] Lustre: DEBUG MARKER: dmesg [ 1635.136783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39c: mtime change on rename =========================================================== 19:09:41 \(1469498981\) [ 1635.351162] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 19:09:41 (1469498981) [ 1637.760673] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1638.203571] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1638.660810] Lustre: DEBUG MARKER: dmesg [ 1639.235779] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39d: create, utime, stat ============================================================== 19:09:46 \(1469498986\) [ 1639.463357] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 19:09:46 (1469498986) [ 1639.756084] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1640.159093] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1640.590913] Lustre: DEBUG MARKER: dmesg [ 1641.070732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39e: create, stat, utime, stat ======================================================== 19:09:47 \(1469498987\) [ 1641.269593] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 19:09:47 (1469498987) [ 1641.579326] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1642.047961] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1642.488726] Lustre: DEBUG MARKER: dmesg [ 1643.027150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39f: create, stat, sleep, utime, stat ================================================= 19:09:49 \(1469498989\) [ 1643.228929] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 19:09:49 (1469498989) [ 1645.486678] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1645.969674] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1646.408397] Lustre: DEBUG MARKER: dmesg [ 1646.963783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39g: write, chmod, stat =============================================================== 19:09:53 \(1469498993\) [ 1647.142363] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 19:09:53 (1469498993) [ 1649.436723] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1649.810510] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1650.298649] Lustre: DEBUG MARKER: dmesg [ 1650.851522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39h: write, utime within one second, stat ============================================= 19:09:57 \(1469498997\) [ 1651.052931] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 19:09:57 (1469498997) [ 1652.381328] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1652.827477] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1653.272171] Lustre: DEBUG MARKER: dmesg [ 1653.799858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39i: write, rename, stat ============================================================== 19:10:00 \(1469499000\) [ 1654.000917] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 19:10:00 (1469499000) [ 1655.256949] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1655.646603] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1656.107402] Lustre: DEBUG MARKER: dmesg [ 1656.525126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39j: write, rename, close, stat ======================================================= 19:10:03 \(1469499003\) [ 1656.712334] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 19:10:03 (1469499003) [ 1656.873045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 1657.197615] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 1657.583862] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 1660.059027] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 1660.668593] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 1661.042340] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1661.419099] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1661.784682] Lustre: DEBUG MARKER: dmesg [ 1662.264550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39k: write, utime, close, stat ======================================================== 19:10:09 \(1469499009\) [ 1662.476864] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 19:10:09 (1469499009) [ 1663.808795] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1664.195354] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1664.576721] Lustre: DEBUG MARKER: dmesg [ 1664.970985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39l: directory atime update =========================================================== 19:10:11 \(1469499011\) [ 1665.171382] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 19:10:11 (1469499011) [ 1665.340207] Lustre: DEBUG MARKER: lctl get_param -n mdd.*MDT0000*.atime_diff [ 1665.625709] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=2 [ 1668.969658] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 1672.331938] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 1672.691392] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1673.123608] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1673.538829] Lustre: DEBUG MARKER: dmesg [ 1674.007700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39m: test atime and mtime before 1970 ================================================= 19:10:20 \(1469499020\) [ 1674.189311] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 19:10:20 (1469499020) [ 1676.530145] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1676.970629] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1677.389984] Lustre: DEBUG MARKER: dmesg [ 1677.866888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39n: check that O_NOATIME is honored ================================================== 19:10:24 \(1469499024\) [ 1678.093577] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ================================================== 19:10:24 (1469499024) [ 1678.269302] Lustre: DEBUG MARKER: lctl get_param -n mdd.*MDT0000*.atime_diff [ 1678.581169] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=1 [ 1688.979545] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 1689.321614] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1689.786111] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1690.181998] Lustre: DEBUG MARKER: dmesg [ 1690.723451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39o: directory cached attributes updated after create ================================= 19:10:37 \(1469499037\) [ 1690.925665] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ================================= 19:10:37 (1469499037) [ 1691.263485] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1691.662793] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1692.121725] Lustre: DEBUG MARKER: dmesg [ 1692.648557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39p: remote directory cached attributes updated after create ========================== 19:10:39 \(1469499039\) [ 1692.829689] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========================== 19:10:39 (1469499039) [ 1693.101639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_39p needs \>= 2 MDTs [ 1693.282519] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [ 1693.500743] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1693.916199] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1694.412127] Lustre: DEBUG MARKER: dmesg [ 1694.954053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39p: close won\'t zero out atime ======================================================= 19:10:41 \(1469499041\) [ 1695.120909] Lustre: DEBUG MARKER: == sanity test 39p: close won't zero out atime ======================================================= 19:10:41 (1469499041) [ 1695.425921] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1695.829650] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1696.272122] Lustre: DEBUG MARKER: dmesg [ 1696.773311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 40: failed open\(O_TRUNC\) doesn\'t truncate ============================================= 19:10:43 \(1469499043\) [ 1696.996267] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 19:10:43 (1469499043) [ 1697.277849] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1697.729919] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1698.169732] Lustre: DEBUG MARKER: dmesg [ 1698.654564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 41: test small file write + fstat ===================================================== 19:10:45 \(1469499045\) [ 1698.832334] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 19:10:45 (1469499045) [ 1700.086027] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1700.547539] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1700.961096] Lustre: DEBUG MARKER: dmesg [ 1701.408709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 1701.602900] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 1701.816305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 1702.016410] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 1702.221660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 1702.420070] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 1702.669648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42d skipping ALWAYS excluded test 42d [ 1702.836543] Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d [ 1703.117582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 19:10:49 \(1469499049\) [ 1703.301963] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 19:10:49 (1469499049) [ 1723.607581] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1724.041381] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1724.397427] Lustre: DEBUG MARKER: dmesg [ 1724.828423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43: execution of file opened for write should return -ETXTBSY ========================= 19:11:11 \(1469499071\) [ 1725.073155] Lustre: DEBUG MARKER: == sanity test 43: execution of file opened for write should return -ETXTBSY ========================= 19:11:11 (1469499071) [ 1726.345293] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1726.795755] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1727.180919] Lustre: DEBUG MARKER: dmesg [ 1727.612329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43a: open\(RDWR\) of file being executed should return -ETXTBSY ========================= 19:11:14 \(1469499074\) [ 1727.793650] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 19:11:14 (1469499074) [ 1728.042580] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1728.432681] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1729.192249] Lustre: DEBUG MARKER: dmesg [ 1729.843689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 19:11:16 \(1469499076\) [ 1730.056464] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 19:11:16 (1469499076) [ 1730.346790] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1730.830591] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1731.312978] Lustre: DEBUG MARKER: dmesg [ 1731.829370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43c: md5sum of copy into lustre======================================================== 19:11:18 \(1469499078\) [ 1732.034214] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre======================================================== 19:11:18 (1469499078) [ 1732.321837] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1732.778841] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1733.214523] Lustre: DEBUG MARKER: dmesg [ 1733.707241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 44: zero length read from a sparse stripe ============================================= 19:11:20 \(1469499080\) [ 1733.910617] Lustre: DEBUG MARKER: == sanity test 44: zero length read from a sparse stripe ============================================= 19:11:20 (1469499080) [ 1734.153539] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1734.626893] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1735.155809] Lustre: DEBUG MARKER: dmesg [ 1735.675062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 44a: test sparse pwrite =============================================================== 19:11:22 \(1469499082\) [ 1735.946053] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 19:11:22 (1469499082) [ 1736.480992] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1736.887436] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1737.356298] Lustre: DEBUG MARKER: dmesg [ 1737.880361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_45 skipping ALWAYS excluded test 45 [ 1738.088058] Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 [ 1738.408893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 46: dirtying a previously written page ================================================ 19:11:25 \(1469499085\) [ 1738.642437] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 19:11:25 (1469499085) [ 1739.332831] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1739.728129] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1740.142321] Lustre: DEBUG MARKER: dmesg [ 1740.639971] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48a: Access renamed working dir \(should return errors\)================================= 19:11:27 \(1469499087\) [ 1740.832972] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 19:11:27 (1469499087) [ 1741.021421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1741.550325] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1741.983362] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1742.351671] Lustre: DEBUG MARKER: dmesg [ 1742.843412] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48b: Access removed working dir \(should return errors\)================================= 19:11:29 \(1469499089\) [ 1743.062229] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 19:11:29 (1469499089) [ 1743.345592] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1743.795207] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1744.241226] Lustre: DEBUG MARKER: dmesg [ 1744.770987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48c: Access removed working subdir \(should return errors\) ============================= 19:11:31 \(1469499091\) [ 1744.952198] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 19:11:31 (1469499091) [ 1745.278083] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1745.707307] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1746.169624] Lustre: DEBUG MARKER: dmesg [ 1746.733691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48d: Access removed parent subdir \(should return errors\) ============================== 19:11:33 \(1469499093\) [ 1746.927101] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 19:11:33 (1469499093) [ 1747.240877] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1747.690758] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1748.093247] Lustre: DEBUG MARKER: dmesg [ 1748.613990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48e: Access to recreated parent subdir \(should return errors\) ========================= 19:11:35 \(1469499095\) [ 1748.782233] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 19:11:35 (1469499095) [ 1749.166421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1749.598762] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1750.039422] Lustre: DEBUG MARKER: dmesg [ 1750.534429] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 49: Change max_pages_per_rpc won\'t break osc extent =================================== 19:11:37 \(1469499097\) [ 1750.729605] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent =================================== 19:11:37 (1469499097) [ 1751.354031] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1751.802857] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1752.281402] Lustre: DEBUG MARKER: dmesg [ 1752.837574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 50: special situations: \/proc symlinks =============================================== 19:11:39 \(1469499099\) [ 1753.049586] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 19:11:39 (1469499099) [ 1753.304733] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1753.697074] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1754.118884] Lustre: DEBUG MARKER: dmesg [ 1754.556831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51a: special situations: split htree with empty entry ================================= 19:11:41 \(1469499101\) [ 1754.752135] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 19:11:41 (1469499101) [ 1755.518277] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1756.132055] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1756.589934] Lustre: DEBUG MARKER: dmesg [ 1757.198591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_51b skipping SLOW test 51b [ 1757.426435] Lustre: DEBUG MARKER: SKIP: sanity test_51b skipping SLOW test 51b [ 1757.693555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51d: check object distribution ======================================================== 19:11:44 \(1469499104\) [ 1757.877717] Lustre: DEBUG MARKER: == sanity test 51d: check object distribution ======================================================== 19:11:44 (1469499104) [ 1758.141035] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_51d skipping test with few OSTs [ 1758.352362] Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping test with few OSTs [ 1758.575666] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1758.998151] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1759.459543] Lustre: DEBUG MARKER: dmesg [ 1759.949533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51e: check file nlink limit =========================================================== 19:11:46 \(1469499106\) [ 1760.143200] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =========================================================== 19:11:46 (1469499106) [ 1760.367955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_51e Only applicable to ldiskfs-based MDTs [ 1760.549796] Lustre: DEBUG MARKER: SKIP: sanity test_51e Only applicable to ldiskfs-based MDTs [ 1760.776523] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1761.223142] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1761.669408] Lustre: DEBUG MARKER: dmesg [ 1762.193880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 52a: append-only flag test \(should return errors\) ===================================== 19:11:49 \(1469499109\) [ 1762.371452] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 19:11:49 (1469499109) [ 1762.829551] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1763.279592] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1763.710625] Lustre: DEBUG MARKER: dmesg [ 1764.192641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 52b: immutable flag test \(should return errors\) ======================================= 19:11:51 \(1469499111\) [ 1764.361670] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 19:11:51 (1469499111) [ 1764.685995] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1765.109564] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1765.529896] Lustre: DEBUG MARKER: dmesg [ 1766.034472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 19:11:52 \(1469499112\) [ 1766.225449] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 19:11:52 (1469499112) [ 1766.479375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1766.840825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1767.192632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1767.520145] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1767.820471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osc.*OST*-osc-MDT0000.prealloc_last_id [ 1768.136724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_seq [ 1768.448891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1769.089968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_seq [ 1769.379780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1770.073633] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1770.468787] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1770.899440] Lustre: DEBUG MARKER: dmesg [ 1771.289821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54a: unix domain socket test ========================================================== 19:11:58 \(1469499118\) [ 1771.498396] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 19:11:58 (1469499118) [ 1771.867514] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1772.269187] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1772.691212] Lustre: DEBUG MARKER: dmesg [ 1773.207503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54b: char device works in lustre ====================================================== 19:11:59 \(1469499119\) [ 1773.433573] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 19:11:59 (1469499119) [ 1773.722059] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1774.142449] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1774.900044] Lustre: DEBUG MARKER: dmesg [ 1775.406685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54c: block device works in lustre ===================================================== 19:12:02 \(1469499122\) [ 1775.604858] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 19:12:02 (1469499122) [ 1784.556995] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1785.017914] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1785.499193] Lustre: DEBUG MARKER: dmesg [ 1785.979602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54d: fifo device works in lustre ====================================================== 19:12:12 \(1469499132\) [ 1786.177526] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 19:12:12 (1469499132) [ 1786.394836] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1786.846289] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1787.278653] Lustre: DEBUG MARKER: dmesg [ 1787.812758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54e: console\/tty device works in lustre =============================================== 19:12:14 \(1469499134\) [ 1788.018898] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 19:12:14 (1469499134) [ 1788.273488] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1788.718441] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1789.121971] Lustre: DEBUG MARKER: dmesg [ 1789.588414] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56a: check \/usr\/bin\/lfs getstripe ===================================================== 19:12:16 \(1469499136\) [ 1789.765924] Lustre: DEBUG MARKER: == sanity test 56a: check /usr/bin/lfs getstripe ===================================================== 19:12:16 (1469499136) [ 1789.809293] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1789.811367] Lustre: Skipped 11 previous similar messages [ 1790.255851] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1790.722322] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1791.142536] Lustre: DEBUG MARKER: dmesg [ 1791.625705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56g: check lfs find -name ============================================================= 19:12:18 \(1469499138\) [ 1791.811130] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 19:12:18 (1469499138) [ 1791.829489] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1791.834715] Lustre: Skipped 5 previous similar messages [ 1792.296621] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1792.714417] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1793.201835] Lustre: DEBUG MARKER: dmesg [ 1793.729726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56h: check lfs find ! -name =========================================================== 19:12:20 \(1469499140\) [ 1793.939168] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 19:12:20 (1469499140) [ 1793.985054] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1793.987208] Lustre: Skipped 5 previous similar messages [ 1794.277842] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1794.663912] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1795.167368] Lustre: DEBUG MARKER: dmesg [ 1795.688583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56i: check \'lfs find -ost UUID\' skips directories ===================================== 19:12:22 \(1469499142\) [ 1795.928989] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 19:12:22 (1469499142) [ 1796.169494] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1796.569086] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1796.999981] Lustre: DEBUG MARKER: dmesg [ 1797.346723] LustreError: 21250:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 1797.474789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56j: check lfs find -type d =========================================================== 19:12:24 \(1469499144\) [ 1797.649040] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 19:12:24 (1469499144) [ 1798.159247] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1798.516384] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1798.947754] Lustre: DEBUG MARKER: dmesg [ 1799.439905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56k: check lfs find -type f =========================================================== 19:12:26 \(1469499146\) [ 1799.639391] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 19:12:26 (1469499146) [ 1800.021267] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1800.408852] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1800.800576] Lustre: DEBUG MARKER: dmesg [ 1801.280568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56l: check lfs find -type b =========================================================== 19:12:28 \(1469499148\) [ 1801.473705] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 19:12:28 (1469499148) [ 1801.784496] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1802.153032] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1802.560826] Lustre: DEBUG MARKER: dmesg [ 1803.013209] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56m: check lfs find -type c =========================================================== 19:12:29 \(1469499149\) [ 1803.197737] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 19:12:29 (1469499149) [ 1803.521206] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1803.899597] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1804.309281] Lustre: DEBUG MARKER: dmesg [ 1804.821482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56n: check lfs find -type l =========================================================== 19:12:31 \(1469499151\) [ 1805.026316] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 19:12:31 (1469499151) [ 1805.539949] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1805.998126] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1806.501985] Lustre: DEBUG MARKER: dmesg [ 1807.001407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56o: check lfs find -mtime for old files ============================================== 19:12:33 \(1469499153\) [ 1807.182579] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 19:12:33 (1469499153) [ 1808.211387] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1808.700197] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1809.145530] Lustre: DEBUG MARKER: dmesg [ 1809.581033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56p: check lfs find -uid and ! -uid =================================================== 19:12:36 \(1469499156\) [ 1809.773621] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 19:12:36 (1469499156) [ 1810.438896] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1810.839180] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1811.248378] Lustre: DEBUG MARKER: dmesg [ 1811.709685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56q: check lfs find -gid and ! -gid =================================================== 19:12:38 \(1469499158\) [ 1811.895607] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 19:12:38 (1469499158) [ 1812.427798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1812.876343] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1813.352283] Lustre: DEBUG MARKER: dmesg [ 1813.866539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56r: check lfs find -size works ======================================================= 19:12:40 \(1469499160\) [ 1814.105686] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 19:12:40 (1469499160) [ 1815.120810] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1815.617512] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1816.077509] Lustre: DEBUG MARKER: dmesg [ 1816.564863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56s: check lfs find -stripe-count works =============================================== 19:12:43 \(1469499163\) [ 1816.754598] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works =============================================== 19:12:43 (1469499163) [ 1817.517727] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1817.984187] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1818.389342] Lustre: DEBUG MARKER: dmesg [ 1818.947056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56t: check lfs find -stripe-size works ================================================ 19:12:45 \(1469499165\) [ 1819.131307] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ================================================ 19:12:45 (1469499165) [ 1820.022928] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1820.512126] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1821.280567] Lustre: DEBUG MARKER: dmesg [ 1821.825053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56u: check lfs find -stripe-index works =============================================== 19:12:48 \(1469499168\) [ 1822.036206] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works =============================================== 19:12:48 (1469499168) [ 1822.830105] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1823.324821] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1823.772561] Lustre: DEBUG MARKER: dmesg [ 1824.299501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56v: check \'lfs find -mdt match with lfs getstripe -M\' ================================ 19:12:51 \(1469499171\) [ 1824.463878] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ================================ 19:12:51 (1469499171) [ 1825.281609] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1825.737715] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1826.151896] Lustre: DEBUG MARKER: dmesg [ 1826.694194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 19:12:53 \(1469499173\) [ 1826.915512] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 19:12:53 (1469499173) [ 1837.530788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1922.626705] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1923.099673] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1923.533094] Lustre: DEBUG MARKER: dmesg [ 1924.055418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56x: lfs migration support ============================================================ 19:14:30 \(1469499270\) [ 1924.219630] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ============================================================ 19:14:30 (1469499270) [ 1939.319332] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1939.725695] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1940.154460] Lustre: DEBUG MARKER: dmesg [ 1940.683560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56xa: lfs migration --block support =================================================== 19:14:47 \(1469499287\) [ 1940.880190] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support =================================================== 19:14:47 (1469499287) [ 1943.388231] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1943.807648] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1944.255108] Lustre: DEBUG MARKER: dmesg [ 1944.772350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56y: lfs find -L raid0\|released ======================================================= 19:14:51 \(1469499291\) [ 1944.955954] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released ======================================================= 19:14:51 (1469499291) [ 1945.149698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1945.568938] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1946.010812] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1946.433996] Lustre: DEBUG MARKER: dmesg [ 1946.964554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56z: lfs find should continue after an error ========================================== 19:14:53 \(1469499293\) [ 1947.204765] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================== 19:14:53 (1469499293) [ 1955.373054] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1955.850057] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1956.251620] Lustre: DEBUG MARKER: dmesg [ 1956.749563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56aa: lfs find --size under striped dir =============================================== 19:15:03 \(1469499303\) [ 1956.962355] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir =============================================== 19:15:03 (1469499303) [ 1957.224953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_56aa needs \>= 2 MDTs [ 1957.411833] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [ 1957.669438] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1958.179075] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1958.593703] Lustre: DEBUG MARKER: dmesg [ 1959.101811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 57a: verify MDS filesystem created with large inodes ================================== 19:15:05 \(1469499305\) [ 1959.292476] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 19:15:05 (1469499305) [ 1959.530902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_57a Only applicable to ldiskfs-based MDTs [ 1959.724171] Lustre: DEBUG MARKER: SKIP: sanity test_57a Only applicable to ldiskfs-based MDTs [ 1959.990682] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1960.395568] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1960.832983] Lustre: DEBUG MARKER: dmesg [ 1961.418140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 19:15:08 \(1469499308\) [ 1961.593874] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 19:15:08 (1469499308) [ 1961.805840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_57b Only applicable to ldiskfs-based MDTs [ 1962.031608] Lustre: DEBUG MARKER: SKIP: sanity test_57b Only applicable to ldiskfs-based MDTs [ 1962.327124] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1962.746410] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1963.267628] Lustre: DEBUG MARKER: dmesg [ 1963.759814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 58: verify cross-platform wire constants ============================================== 19:15:10 \(1469499310\) [ 1963.966367] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 19:15:10 (1469499310) [ 1964.217859] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1964.623691] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1965.009901] Lustre: DEBUG MARKER: dmesg [ 1965.540681] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 59: verify cancellation of llog records async ========================================= 19:15:12 \(1469499312\) [ 1965.752564] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 19:15:12 (1469499312) [ 1967.550292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 1967.861549] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 1969.288254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 1970.639975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 1971.991228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 1973.330446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 1974.718046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 1981.169671] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1981.613426] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 1982.108717] Lustre: DEBUG MARKER: dmesg [ 1982.834802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 19:15:29 \(1469499329\) [ 1983.089653] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 19:15:29 (1469499329) [ 1983.293438] Lustre: DEBUG MARKER: ! which run-llog.sh &> /dev/null [ 1983.741084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_60 run 9470 - from kernel mode [ 1983.922409] Lustre: DEBUG MARKER: test_60 run 9470 - from kernel mode [ 1984.080853] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=warning; /usr/sbin/lctl dk > /dev/null [ 1984.629905] Lustre: DEBUG MARKER: sh run-llog.sh [ 1985.005835] Lustre: 28935:0:(llog_test.c:2086:llog_test_setup()) Setup llog-test device over MGS device [ 1985.010167] Lustre: 28935:0:(llog_test.c:115:llog_test_1()) 1a: create a log with name: e0ba8587 [ 1985.012737] Lustre: 28935:0:(llog_test.c:132:llog_test_1()) 1b: close newly-created log [ 1985.014517] Lustre: 28935:0:(llog_test.c:163:llog_test_2()) 2a: re-open a log with name: e0ba8587 [ 1985.016763] Lustre: 28935:0:(llog_test.c:183:llog_test_2()) 2b: create a log without specified NAME & LOGID [ 1985.018902] Lustre: 28935:0:(llog_test.c:201:llog_test_2()) 2b: write 1 llog records, check llh_count [ 1985.021118] Lustre: 28935:0:(llog_test.c:214:llog_test_2()) 2c: re-open the log by LOGID and verify llh_count [ 1985.023849] Lustre: 28935:0:(llog_test.c:261:llog_test_2()) 2d: destroy this log [ 1985.025713] Lustre: 28935:0:(llog_test.c:415:llog_test_3()) 3a: write 1023 fixed-size llog records [ 1985.049261] Lustre: 28935:0:(llog_test.c:381:llog_test3_process()) test3: processing records from index 501 to the end [ 1985.058873] Lustre: 28935:0:(llog_test.c:390:llog_test3_process()) test3: total 525 records processed with 0 paddings [ 1985.060953] Lustre: 28935:0:(llog_test.c:471:llog_test_3()) 3b: write 566 variable size llog records [ 1985.086381] Lustre: 28935:0:(llog_test.c:540:llog_test_3()) 3c: write records with variable size until BITMAP_SIZE, return -ENOSPC [ 1986.444782] Lustre: 28935:0:(llog_test.c:563:llog_test_3()) 3c: wrote 63962 more records before end of llog is reached [ 1986.447222] Lustre: 28935:0:(llog_test.c:592:llog_test_4()) 4a: create a catalog log with name: e0ba8588 [ 1986.449750] Lustre: 28935:0:(llog_test.c:607:llog_test_4()) 4b: write 1 record into the catalog [ 1986.452329] Lustre: 28935:0:(llog_test.c:622:llog_test_4()) 4c: cancel 1 log record [ 1986.454411] Lustre: 28935:0:(llog_test.c:634:llog_test_4()) 4d: write 65536 more log records [ 1990.708533] Lustre: 28935:0:(llog_test.c:650:llog_test_4()) 4e: add 5 large records, one record per block [ 1990.711246] Lustre: 28935:0:(llog_test.c:670:llog_test_4()) 4f: put newly-created catalog [ 1990.713142] Lustre: 28935:0:(llog_test.c:768:llog_test_5()) 5a: re-open catalog by id [ 1990.715132] Lustre: 28935:0:(llog_test.c:781:llog_test_5()) 5b: print the catalog entries.. we expect 2 [ 1990.717388] Lustre: 28942:0:(llog_test.c:699:cat_print_cb()) seeing record at index 1 - [0x1:0xf:0x0] in log [0xa:0xf:0x0] [ 1990.719364] Lustre: 28935:0:(llog_test.c:793:llog_test_5()) 5c: Cancel 65536 records, see one log zapped [ 2018.348867] Lustre: 28935:0:(llog_test.c:801:llog_test_5()) 5c: print the catalog entries.. we expect 1 [ 2018.351363] Lustre: 28943:0:(llog_test.c:699:cat_print_cb()) seeing record at index 2 - [0x1:0x10:0x0] in log [0xa:0xf:0x0] [ 2018.357339] Lustre: 28943:0:(llog_test.c:699:cat_print_cb()) Skipped 1 previous similar message [ 2018.360502] Lustre: 28935:0:(llog_test.c:813:llog_test_5()) 5d: add 1 record to the log with many canceled empty pages [ 2018.366539] Lustre: 28935:0:(llog_test.c:821:llog_test_5()) 5e: print plain log entries.. expect 6 [ 2018.370603] Lustre: 28935:0:(llog_test.c:833:llog_test_5()) 5f: print plain log entries reversely.. expect 6 [ 2018.373533] Lustre: 28935:0:(llog_test.c:847:llog_test_5()) 5g: close re-opened catalog [ 2018.375372] Lustre: 28935:0:(llog_test.c:877:llog_test_6()) 6a: re-open log e0ba8587 using client API [ 2018.377442] Lustre: MGS: non-config logname received: e0ba8587 [ 2018.379418] Lustre: 28935:0:(llog_test.c:909:llog_test_6()) 6b: process log e0ba8587 using client API [ 2018.487474] Lustre: 28935:0:(llog_test.c:913:llog_test_6()) 6b: processed 63962 records [ 2018.489771] Lustre: 28935:0:(llog_test.c:920:llog_test_6()) 6c: process log e0ba8587 reversely using client API [ 2023.968198] Lustre: 28935:0:(llog_test.c:924:llog_test_6()) 6c: processed 63962 records [ 2023.970611] Lustre: 28935:0:(llog_test.c:1071:llog_test_7()) 7a: test llog_logid_rec [ 2082.774160] Lustre: 28935:0:(llog_test.c:1082:llog_test_7()) 7b: test llog_unlink64_rec [ 2152.823254] Lustre: 28935:0:(llog_test.c:1093:llog_test_7()) 7c: test llog_setattr64_rec [ 2224.212275] Lustre: 28935:0:(llog_test.c:1104:llog_test_7()) 7d: test llog_size_change_rec [ 2236.717039] Lustre: 28935:0:(llog_test.c:1115:llog_test_7()) 7e: test llog_changelog_rec [ 2238.003777] Lustre: 28935:0:(llog_test.c:1021:llog_test_7_sub()) 7_sub: records are not aligned, written 64071 from 64767 [ 2319.627482] Lustre: 28935:0:(llog_test.c:1127:llog_test_7()) 7f: test llog_changelog_user_rec [ 2320.884503] Lustre: 28935:0:(llog_test.c:1021:llog_test_7_sub()) 7_sub: records are not aligned, written 64452 from 64767 [ 2353.210062] Lustre: 28935:0:(llog_test.c:1138:llog_test_7()) 7g: test llog_gen_rec [ 2392.644111] Lustre: 28935:0:(llog_test.c:1233:llog_test_8()) 8a: fill the first plain llog [ 2392.653621] Lustre: 28935:0:(llog_test.c:1266:llog_test_8()) 8a: pin llog [0x1:0x1a:0x0] [ 2392.655446] Lustre: 28935:0:(llog_test.c:1276:llog_test_8()) 8b: fill the second plain llog [ 2392.664406] Lustre: 28935:0:(llog_test.c:1297:llog_test_8()) 8b: second llog [0x1:0x1c:0x0] [ 2392.666127] Lustre: 28935:0:(llog_test.c:1307:llog_test_8()) 8c: drop two records from the first plain llog [ 2392.668191] Lustre: 28935:0:(llog_test.c:1310:llog_test_8()) 8d: count survived records [ 2392.670334] LustreError: 28935:0:(llog.c:604:llog_process_thread()) Local llog found corrupted [ 2392.672486] Lustre: 28935:0:(llog_test.c:1337:llog_test_8()) 8d: close re-opened catalog [ 2392.674239] Lustre: 28935:0:(llog_test.c:1399:llog_test_9()) 9a: test llog_logid_rec [ 2392.676247] Lustre: 28935:0:(llog_test.c:1383:llog_test_9_sub()) 9_sub: record type 1064553b in log 0x1:0x1e:0x0 [ 2392.678125] Lustre: 28935:0:(llog_test.c:1410:llog_test_9()) 9b: test llog_obd_cfg_rec [ 2392.680127] Lustre: 28935:0:(llog_test.c:1421:llog_test_9()) 9c: test llog_changelog_rec [ 2392.681999] Lustre: 28935:0:(llog_test.c:1433:llog_test_9()) 9d: test llog_changelog_user_rec [ 2392.683952] Lustre: 28935:0:(llog_test.c:1470:llog_test_10()) 10a: create a catalog log with name: e0ba8589 [ 2398.169427] Lustre: 28935:0:(llog_test.c:1498:llog_test_10()) 10b: write 65536 log records [ 2404.116370] Lustre: 28935:0:(llog_test.c:1522:llog_test_10()) 10c: write 131072 more log records [ 2420.568664] Lustre: 28935:0:(llog_test.c:1550:llog_test_10()) 10c: write 65536 more log records [ 2425.462304] Lustre: 28935:0:(llog_cat.c:93:llog_cat_new_log()) MGS: there are no more free slots in catalog [ 2425.714366] Lustre: 28935:0:(llog_test.c:1575:llog_test_10()) 10c: wrote 61704 records then 3832 failed with ENOSPC [ 2425.716635] Lustre: 28935:0:(llog_test.c:1592:llog_test_10()) 10d: Cancel 65536 records, see one log zapped [ 2455.032454] Lustre: 28935:0:(llog_test.c:1604:llog_test_10()) 10d: print the catalog entries.. we expect 3 [ 2455.034731] Lustre: 28976:0:(llog_test.c:699:cat_print_cb()) seeing record at index 2 - [0x1:0x23:0x0] in log [0xa:0x10:0x0] [ 2460.274263] Lustre: 28935:0:(llog_test.c:1632:llog_test_10()) 10e: write 65536 more log records [ 2464.853534] Lustre: 28935:0:(llog_cat.c:93:llog_cat_new_log()) MGS: there are no more free slots in catalog [ 2464.855770] Lustre: 28935:0:(llog_cat.c:93:llog_cat_new_log()) Skipped 3831 previous similar messages [ 2464.917584] Lustre: 28935:0:(llog_test.c:1657:llog_test_10()) 10e: wrote 64578 records then 958 failed with ENOSPC [ 2464.919494] Lustre: 28935:0:(llog_test.c:1659:llog_test_10()) 10e: print the catalog entries.. we expect 4 [ 2464.921632] Lustre: 28977:0:(llog_test.c:699:cat_print_cb()) seeing record at index 1 - [0x1:0xf1e:0x0] in log [0xa:0x10:0x0] [ 2464.923637] Lustre: 28977:0:(llog_test.c:699:cat_print_cb()) Skipped 2 previous similar messages [ 2464.925728] Lustre: 28935:0:(llog_test.c:1695:llog_test_10()) 10e: catalog successfully wrap around, last_idx 1, first 1 [ 2466.244949] Lustre: 28935:0:(llog_test.c:1708:llog_test_10()) 10f: Cancel 65536 records, see one log zapped [ 2466.250790] Lustre: 28935:0:(llog_cat.c:817:llog_cat_process_or_fork()) catlog 0x10:10 crosses index zero [ 2495.840161] Lustre: 28935:0:(llog_test.c:1720:llog_test_10()) 10f: print the catalog entries.. we expect 3 [ 2495.842573] Lustre: 28980:0:(llog_test.c:699:cat_print_cb()) seeing record at index 1 - [0x1:0xf1e:0x0] in log [0xa:0x10:0x0] [ 2495.844682] Lustre: 28980:0:(llog_test.c:699:cat_print_cb()) Skipped 3 previous similar messages [ 2496.899586] Lustre: 28935:0:(llog_test.c:1748:llog_test_10()) 10f: write 65536 more log records [ 2501.486365] Lustre: 28935:0:(llog_cat.c:93:llog_cat_new_log()) MGS: there are no more free slots in catalog [ 2501.488585] Lustre: 28935:0:(llog_cat.c:93:llog_cat_new_log()) Skipped 957 previous similar messages [ 2501.547173] Lustre: 28935:0:(llog_test.c:1773:llog_test_10()) 10f: wrote 64578 records then 958 failed with ENOSPC [ 2502.887623] Lustre: 28935:0:(llog_test.c:1816:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 2502.891762] Lustre: 28935:0:(llog_cat.c:817:llog_cat_process_or_fork()) catlog 0x10:10 crosses index zero [ 2532.301226] Lustre: 28935:0:(llog_test.c:1828:llog_test_10()) 10g: print the catalog entries.. we expect 3 [ 2532.303723] Lustre: 28983:0:(llog_test.c:699:cat_print_cb()) seeing record at index 1 - [0x1:0xf1e:0x0] in log [0xa:0x10:0x0] [ 2532.309811] Lustre: 28983:0:(llog_test.c:699:cat_print_cb()) Skipped 2 previous similar messages [ 2533.644549] Lustre: 28935:0:(llog_test.c:1855:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 2533.647777] Lustre: 28935:0:(llog_cat.c:817:llog_cat_process_or_fork()) catlog 0x10:10 crosses index zero [ 2563.441813] Lustre: 28935:0:(llog_test.c:1867:llog_test_10()) 10g: print the catalog entries.. we expect 2 [ 2563.444287] Lustre: 28986:0:(llog_test.c:699:cat_print_cb()) seeing record at index 1 - [0x1:0xf1e:0x0] in log [0xa:0x10:0x0] [ 2563.446535] Lustre: 28986:0:(llog_test.c:699:cat_print_cb()) Skipped 2 previous similar messages [ 2564.813309] Lustre: 28935:0:(llog_test.c:1902:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 2597.443721] Lustre: 28935:0:(llog_test.c:1914:llog_test_10()) 10g: print the catalog entries.. we expect 1 [ 2597.446977] Lustre: 28989:0:(llog_test.c:699:cat_print_cb()) seeing record at index 2 - [0x1:0x12dd:0x0] in log [0xa:0x10:0x0] [ 2597.452042] Lustre: 28989:0:(llog_test.c:699:cat_print_cb()) Skipped 1 previous similar message [ 2597.454296] Lustre: 28935:0:(llog_test.c:1939:llog_test_10()) 10g: llh_cat_idx has also successfully wrapped! [ 2597.457815] Lustre: 28935:0:(llog_test.c:1945:llog_test_10()) 10: put newly-created catalog [ 2597.630545] Lustre: DEBUG MARKER: /usr/sbin/lctl dk [ 2598.110490] Lustre: DEBUG MARKER: which llog_reader 2> /dev/null [ 2598.443077] Lustre: DEBUG MARKER: ls -d /sbin/llog_reader [ 2598.849721] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2599.122335] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2599.264393] Lustre: Failing over lustre-MDT0000 [ 2601.494786] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.87@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.499225] LustreError: Skipped 7 previous similar messages [ 2603.824061] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.85@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.831835] LustreError: Skipped 1 previous similar message [ 2606.453164] Lustre: 29337:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469499946/real 1469499946] req@ffff880049ac6700 x1540879853690080/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469499952 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2606.932193] Lustre: server umount lustre-MDT0000 complete [ 2607.068824] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2607.391809] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 2609.380784] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2612.582562] Lustre: DEBUG MARKER: zfs get -H -o value canmount lustre-mdt1/mdt1 [ 2612.900163] Lustre: DEBUG MARKER: zfs get -H -o value mountpoint lustre-mdt1/mdt1 [ 2613.192263] Lustre: DEBUG MARKER: zfs set canmount=noauto lustre-mdt1/mdt1 [ 2615.489799] Lustre: DEBUG MARKER: zfs set mountpoint=legacy lustre-mdt1/mdt1 [ 2617.534205] Lustre: DEBUG MARKER: mount -t zfs lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2617.899523] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1e:0x0 [ 2618.229079] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1e:0x0 [ 2618.540470] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1e:0x0 [ 2618.882517] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1f:0x0 [ 2619.180652] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1f:0x0 [ 2619.510828] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x20:0x0 [ 2619.810337] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x20:0x0 [ 2620.110567] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x21:0x0 [ 2620.382540] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x21:0x0 [ 2620.742808] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2621.088715] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2621.411064] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2621.791322] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 2625.380694] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2625.750069] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2632.398121] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2632.714421] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2632.912045] Lustre: MGS: Connection restored to f14df7fe-c07f-3b6c-2e32-0b0fa5cf5034 (at 0@lo) [ 2632.915059] Lustre: Skipped 11 previous similar messages [ 2634.897047] LustreError: 769:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 2636.846998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2637.133297] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2637.621339] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2637.898060] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2638.194923] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2638.527479] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2639.059988] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2639.490952] Lustre: DEBUG MARKER: dmesg [ 2639.510671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2640.520183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60b: limit repeated messages from CERROR\/CWARN ======================================== 19:26:27 \(1469499987\) [ 2640.752359] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 19:26:27 (1469499987) [ 2646.331683] Lustre: lustre-MDT0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 2646.571850] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2646.969317] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2647.440396] Lustre: DEBUG MARKER: dmesg [ 2648.112467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60c: unlink file when mds full ======================================================== 19:26:34 \(1469499994\) [ 2648.299669] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 19:26:34 (1469499994) [ 2690.309879] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2690.693918] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2691.071643] Lustre: DEBUG MARKER: dmesg [ 2691.611859] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60d: test printk console message masking ============================================== 19:27:18 \(1469500038\) [ 2691.802804] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 19:27:18 (1469500038) [ 2692.075342] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2692.473267] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2692.861635] Lustre: DEBUG MARKER: dmesg [ 2693.389299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60e: no space while new llog is being created ========================================= 19:27:20 \(1469500040\) [ 2693.638282] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================= 19:27:20 (1469500040) [ 2693.860992] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x15b [ 2694.019492] Lustre: *** cfs_fail_loc=15b, val=0*** [ 2694.021889] Lustre: *** cfs_fail_loc=15b, val=0*** [ 2694.256294] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2694.696287] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2695.115837] Lustre: DEBUG MARKER: dmesg [ 2695.597403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 61: mmap\(\) writes don\'t make sync hang ================================================ 19:27:22 \(1469500042\) [ 2695.814343] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 19:27:22 (1469500042) [ 2696.638034] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2697.006726] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2697.521452] Lustre: DEBUG MARKER: dmesg [ 2697.991911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 19:27:24 \(1469500044\) [ 2698.248098] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 19:27:24 (1469500044) [ 2758.587548] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2759.038733] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2759.491352] Lustre: DEBUG MARKER: dmesg [ 2759.927357] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 63b: async write errors should be returned to fsync =================================== 19:28:26 \(1469500106\) [ 2760.102854] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 19:28:26 (1469500106) [ 2760.311756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2765.708748] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2766.150963] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2766.542932] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2766.962864] Lustre: DEBUG MARKER: dmesg [ 2767.408570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 64a: verify filter grant calculations \(in kernel\) ===================================== 19:28:34 \(1469500114\) [ 2767.627531] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 19:28:34 (1469500114) [ 2767.848756] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2768.272611] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2768.661283] Lustre: DEBUG MARKER: dmesg [ 2769.153483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_64b skipping SLOW test 64b [ 2769.346840] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 2769.568023] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 64c: verify grant shrink ========================------ =============================== 19:28:36 \(1469500116\) [ 2769.782617] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ =============================== 19:28:36 (1469500116) [ 2769.989756] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2770.419822] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2770.859569] Lustre: DEBUG MARKER: dmesg [ 2771.398349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65a: directory with no stripe info ==================================================== 19:28:38 \(1469500118\) [ 2771.628907] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ==================================================== 19:28:38 (1469500118) [ 2771.936103] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2772.360221] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2772.788440] Lustre: DEBUG MARKER: dmesg [ 2773.319071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 19:28:40 \(1469500120\) [ 2773.495184] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 19:28:40 (1469500120) [ 2773.772423] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2774.165796] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2774.578668] Lustre: DEBUG MARKER: dmesg [ 2775.043664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 =================================== 19:28:41 \(1469500121\) [ 2775.239060] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 =================================== 19:28:41 (1469500121) [ 2775.459858] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2775.895249] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2776.271672] Lustre: DEBUG MARKER: dmesg [ 2776.753986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 19:28:43 \(1469500123\) [ 2776.951149] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 19:28:43 (1469500123) [ 2777.282646] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2777.702805] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2778.150607] Lustre: DEBUG MARKER: dmesg [ 2778.661624] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65e: directory setstripe defaults ===================================================== 19:28:45 \(1469500125\) [ 2778.862121] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ===================================================== 19:28:45 (1469500125) [ 2779.200190] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2779.729212] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2780.165387] Lustre: DEBUG MARKER: dmesg [ 2780.657860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65f: dir setstripe permission \(should return error\) =================================== 19:28:47 \(1469500127\) [ 2780.859477] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 19:28:47 (1469500127) [ 2781.421165] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2781.854164] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2782.276866] Lustre: DEBUG MARKER: dmesg [ 2782.809609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65g: directory setstripe -d =========================================================== 19:28:49 \(1469500129\) [ 2783.026255] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 19:28:49 (1469500129) [ 2783.391812] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2783.769920] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2784.315311] Lustre: DEBUG MARKER: dmesg [ 2784.847226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65h: directory stripe info inherit ==================================================== 19:28:51 \(1469500131\) [ 2785.164972] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 19:28:51 (1469500131) [ 2785.461812] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2785.982510] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2786.518434] Lustre: DEBUG MARKER: dmesg [ 2787.070912] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65i: set non-default striping on root directory \(bug 6367\)============================= 19:28:53 \(1469500133\) [ 2787.280998] Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)============================= 19:28:53 (1469500133) [ 2787.296785] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 2787.301847] Lustre: Skipped 5 previous similar messages [ 2787.559806] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2787.980814] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2788.457660] Lustre: DEBUG MARKER: dmesg [ 2789.203490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65ia: getstripe on -1 default directory striping ====================================== 19:28:55 \(1469500135\) [ 2789.407114] Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping ====================================== 19:28:55 (1469500135) [ 2790.625020] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2791.046540] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2791.513750] Lustre: DEBUG MARKER: dmesg [ 2792.037757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 19:28:58 \(1469500138\) [ 2792.225305] Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 19:28:58 (1469500138) [ 2793.402511] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2793.831462] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2794.269742] Lustre: DEBUG MARKER: dmesg [ 2794.678932] LustreError: 7387:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripecount': rc = -34 [ 2794.730066] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_65ic skipping ALWAYS excluded test 65ic [ 2794.921084] Lustre: DEBUG MARKER: SKIP: sanity test_65ic skipping ALWAYS excluded test 65ic [ 2795.204059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65j: set default striping on root directory \(bug 6367\)================================= 19:29:01 \(1469500141\) [ 2795.404628] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 19:29:01 (1469500141) [ 2796.910507] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 2796.916167] Lustre: Skipped 5 previous similar messages [ 2797.166579] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2797.633978] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2798.021298] Lustre: DEBUG MARKER: dmesg [ 2798.564164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 19:29:05 \(1469500145\) [ 2798.751375] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 19:29:05 (1469500145) [ 2798.949922] Lustre: DEBUG MARKER: lctl dl [ 2799.251415] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 activate [ 2799.392306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.2.5.85@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2799.396429] Lustre: Skipped 1 previous similar message [ 2799.411368] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 10.2.5.85@tcp (at 10.2.5.85@tcp) [ 2799.416885] Lustre: Skipped 8 previous similar messages [ 2799.571187] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 2799.878657] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 deactivate [ 2800.022050] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2800.199612] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 2800.581154] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 2800.942666] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 activate [ 2801.072173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.2.5.85@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2801.080467] Lustre: Skipped 1 previous similar message [ 2801.090762] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.239120] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 deactivate [ 2801.400551] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2801.572509] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 2801.901532] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 2802.228960] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 2802.385754] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2802.613858] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2803.057593] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2803.516402] Lustre: DEBUG MARKER: dmesg [ 2804.006633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65l: lfs find on -1 stripe dir ======================================================== 19:29:10 \(1469500150\) [ 2804.203617] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 19:29:10 (1469500150) [ 2804.505955] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2804.961266] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2805.369925] Lustre: DEBUG MARKER: dmesg [ 2805.884774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 66: update inode blocks count on client =============================================== 19:29:12 \(1469500152\) [ 2806.111881] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 19:29:12 (1469500152) [ 2806.404166] LustreError: 9132:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 2806.411232] LustreError: 9132:0:(obd_config.c:1397:class_process_proc_param()) Skipped 1 previous similar message [ 2806.654078] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 2820.664429] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 2821.504792] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2821.936125] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2822.358666] Lustre: DEBUG MARKER: dmesg [ 2822.831851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_68a skipping SLOW test 68a \(base 68\) [ 2823.008046] Lustre: DEBUG MARKER: SKIP: sanity test_68a skipping SLOW test 68a (base 68) [ 2823.237000] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_68b skipping ALWAYS excluded test 68b [ 2823.431222] Lustre: DEBUG MARKER: SKIP: sanity test_68b skipping ALWAYS excluded test 68b [ 2823.700697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 69: verify oa2dentry return -ENOENT doesn\'t LBUG ====================================== 19:29:30 \(1469500170\) [ 2823.917353] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 19:29:30 (1469500170) [ 2829.584108] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2830.101037] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2830.506416] Lustre: DEBUG MARKER: dmesg [ 2831.030861] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_71 skipping SLOW test 71 [ 2831.224157] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 2831.499827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 72a: Test that remove suid works properly \(bug5695\) =================================== 19:29:38 \(1469500178\) [ 2831.725273] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 19:29:38 (1469500178) [ 2834.350902] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2834.776078] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2835.230415] Lustre: DEBUG MARKER: dmesg [ 2835.793714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 72b: Test that we keep mode setting if without file data changed \(bug 24226\) ========== 19:29:42 \(1469500182\) [ 2836.004821] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 19:29:42 (1469500182) [ 2845.278563] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2845.724720] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2846.221192] Lustre: DEBUG MARKER: dmesg [ 2846.789411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 73: multiple MDC requests \(should not deadlock\) ======================================= 19:29:53 \(1469500193\) [ 2846.982740] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 19:29:53 (1469500193) [ 2873.359857] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2873.739538] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2874.130635] Lustre: DEBUG MARKER: dmesg [ 2874.714954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74a: ldlm_enqueue freed-export error path, ls \(shouldn\'t LBUG\) ======================== 19:30:21 \(1469500221\) [ 2874.944520] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 19:30:21 (1469500221) [ 2875.258113] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2875.767069] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2876.182502] Lustre: DEBUG MARKER: dmesg [ 2876.702471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74b: ldlm_enqueue freed-export error path, touch \(shouldn\'t LBUG\) ===================== 19:30:23 \(1469500223\) [ 2876.910025] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 19:30:23 (1469500223) [ 2877.189680] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2877.661063] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2878.052793] Lustre: DEBUG MARKER: dmesg [ 2878.611642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74c: ldlm_lock_create error path, \(shouldn\'t LBUG\) ==================================== 19:30:25 \(1469500225\) [ 2878.811190] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 19:30:25 (1469500225) [ 2879.068839] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2879.485190] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2879.918538] Lustre: DEBUG MARKER: dmesg [ 2880.411562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_76 skipping ALWAYS excluded test 76 [ 2880.621809] Lustre: DEBUG MARKER: SKIP: sanity test_76 skipping ALWAYS excluded test 76 [ 2880.947842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77a: normal checksum read\/write operation ============================================= 19:30:27 \(1469500227\) [ 2881.171674] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 19:30:27 (1469500227) [ 2882.074953] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2882.798544] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2883.289657] Lustre: DEBUG MARKER: dmesg [ 2883.867989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77b: checksum error on client write, read ============================================= 19:30:30 \(1469500230\) [ 2884.093153] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ============================================= 19:30:30 (1469500230) [ 2885.353759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32 [ 2885.587202] Lustre: DEBUG MARKER: set checksum type to crc32 [ 2887.192755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to adler [ 2887.463819] Lustre: DEBUG MARKER: set checksum type to adler [ 2889.071421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to adler [ 2889.269176] Lustre: DEBUG MARKER: set checksum type to adler [ 2889.530821] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2890.050342] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2890.525788] Lustre: DEBUG MARKER: dmesg [ 2891.157560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77d: checksum error on OST direct write, read ========================================= 19:30:37 \(1469500237\) [ 2891.350714] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================= 19:30:37 (1469500237) [ 2896.541020] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2896.948220] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 2897.336649] Lustre: DEBUG MARKER: dmesg [ 2897.873957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77f: repeat checksum error on write \(expect error\) ==================================== 19:30:44 \(1469500244\) [ 2898.065688] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 19:30:44 (1469500244) [ 2898.333786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32 [ 2898.531140] Lustre: DEBUG MARKER: set checksum type to crc32 [ 2967.576926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to adler [ 2967.780654] Lustre: DEBUG MARKER: set checksum type to adler [ 3035.284792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to adler [ 3035.473080] Lustre: DEBUG MARKER: set checksum type to adler [ 3035.702158] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3036.136825] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3036.587725] Lustre: DEBUG MARKER: dmesg [ 3037.310580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77g: checksum error on OST write, read ================================================ 19:33:04 \(1469500384\) [ 3037.499576] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ================================================ 19:33:04 (1469500384) [ 3042.890156] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3043.336374] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3043.762413] Lustre: DEBUG MARKER: dmesg [ 3044.242812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77i: client not supporting OSD_CONNECT_CKSUM ========================================== 19:33:11 \(1469500391\) [ 3044.461646] Lustre: DEBUG MARKER: == sanity test 77i: client not supporting OSD_CONNECT_CKSUM ========================================== 19:33:11 (1469500391) [ 3046.003141] Lustre: MGS: Connection restored to 63beb10b-1bf2-63ca-13cd-729944405c6c (at 10.2.5.86@tcp) [ 3046.008578] Lustre: Skipped 3 previous similar messages [ 3046.484733] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3046.932678] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3047.377095] Lustre: DEBUG MARKER: dmesg [ 3047.908720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77j: client only supporting ADLER32 =================================================== 19:33:14 \(1469500394\) [ 3048.116389] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 19:33:14 (1469500394) [ 3050.916835] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3051.321446] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3051.807205] Lustre: DEBUG MARKER: dmesg [ 3052.333718] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 78: handle large O_DIRECT writes correctly ============================================ 19:33:19 \(1469500399\) [ 3052.554080] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 19:33:19 (1469500399) [ 3091.521447] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3091.951140] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3092.385455] Lustre: DEBUG MARKER: dmesg [ 3092.917195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 79: df report consistency check ======================================================= 19:33:59 \(1469500439\) [ 3093.113684] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 19:33:59 (1469500439) [ 3093.309842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3093.611599] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3099.310302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3100.661000] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3102.021810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3103.345719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3104.690039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3106.069508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3107.392137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3113.833783] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3114.285825] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3114.720511] Lustre: DEBUG MARKER: dmesg [ 3115.210730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 80: Page eviction is equally fast at high offsets too ================================ 19:34:22 \(1469500462\) [ 3115.390269] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 19:34:22 (1469500462) [ 3117.411911] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3117.853179] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3118.353150] Lustre: DEBUG MARKER: dmesg [ 3118.911246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 19:34:25 \(1469500465\) [ 3119.110535] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 19:34:25 (1469500465) [ 3122.649663] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3123.091127] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3123.496372] Lustre: DEBUG MARKER: dmesg [ 3124.104519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 19:34:30 \(1469500470\) [ 3124.316444] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 19:34:30 (1469500470) [ 3129.150790] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3129.535554] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3129.996089] Lustre: DEBUG MARKER: dmesg [ 3130.536004] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 82: Basic grouplock test ============================================================== 19:34:37 \(1469500477\) [ 3130.721624] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ============================================================== 19:34:37 (1469500477) [ 3133.307827] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3133.785566] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3134.255100] Lustre: DEBUG MARKER: dmesg [ 3134.752550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99a: cvs init ========================================================================= 19:34:41 \(1469500481\) [ 3134.970157] Lustre: DEBUG MARKER: == sanity test 99a: cvs init ========================================================================= 19:34:41 (1469500481) [ 3139.147910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3139.524891] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3139.993702] Lustre: DEBUG MARKER: dmesg [ 3140.452514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99b: cvs import ======================================================================= 19:34:47 \(1469500487\) [ 3140.668509] Lustre: DEBUG MARKER: == sanity test 99b: cvs import ======================================================================= 19:34:47 (1469500487) [ 3141.161077] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3141.653969] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3142.153212] Lustre: DEBUG MARKER: dmesg [ 3142.721633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99c: cvs checkout ===================================================================== 19:34:49 \(1469500489\) [ 3142.937025] Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ===================================================================== 19:34:49 (1469500489) [ 3146.247831] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3146.665803] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3147.077126] Lustre: DEBUG MARKER: dmesg [ 3147.674582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99d: cvs add ========================================================================== 19:34:54 \(1469500494\) [ 3147.888584] Lustre: DEBUG MARKER: == sanity test 99d: cvs add ========================================================================== 19:34:54 (1469500494) [ 3148.232292] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3148.654727] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3149.051421] Lustre: DEBUG MARKER: dmesg [ 3149.547978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99e: cvs update ======================================================================= 19:34:56 \(1469500496\) [ 3149.728466] Lustre: DEBUG MARKER: == sanity test 99e: cvs update ======================================================================= 19:34:56 (1469500496) [ 3150.138715] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3150.612439] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3151.006467] Lustre: DEBUG MARKER: dmesg [ 3151.557392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99f: cvs commit ======================================================================= 19:34:58 \(1469500498\) [ 3151.746957] Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ======================================================================= 19:34:58 (1469500498) [ 3157.281207] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3157.687168] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3158.115227] Lustre: DEBUG MARKER: dmesg [ 3158.713488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 100: check local port using privileged port =========================================== 19:35:05 \(1469500505\) [ 3158.880820] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 19:35:05 (1469500505) [ 3159.895870] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3160.317978] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3160.752790] Lustre: DEBUG MARKER: dmesg [ 3161.218371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101a: check read-ahead for random reads =============================================== 19:35:08 \(1469500508\) [ 3161.458087] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads =============================================== 19:35:08 (1469500508) [ 3195.211346] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3195.657306] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3196.091279] Lustre: DEBUG MARKER: dmesg [ 3196.648885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101b: check stride-io mode read-ahead ================================================= 19:35:43 \(1469500543\) [ 3196.839710] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 19:35:43 (1469500543) [ 3205.336812] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3205.759216] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3206.160358] Lustre: DEBUG MARKER: dmesg [ 3206.702682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101c: check stripe_size aligned read-ahead ============================================ 19:35:53 \(1469500553\) [ 3206.898005] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ============================================ 19:35:53 (1469500553) [ 3220.090733] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3220.525900] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3220.991992] Lustre: DEBUG MARKER: dmesg [ 3221.538124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101d: file read with and without read-ahead enabled =================================== 19:36:08 \(1469500568\) [ 3221.785154] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================================== 19:36:08 (1469500568) [ 3507.128868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3507.420580] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3509.751379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3511.078087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3512.418486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3513.788868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3515.144595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3516.530398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3517.849165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3524.305486] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3524.762676] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3525.240736] Lustre: DEBUG MARKER: dmesg [ 3526.046703] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101e: check read-ahead for small read\(1k\) for small files\(500k\) ======================= 19:41:12 \(1469500872\) [ 3526.254311] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ======================= 19:41:12 (1469500872) [ 3534.647265] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3535.190568] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3535.663505] Lustre: DEBUG MARKER: dmesg [ 3536.196538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101f: check mmap read performance ===================================================== 19:41:22 \(1469500882\) [ 3536.420510] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ===================================================== 19:41:22 (1469500882) [ 3555.322549] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3555.747708] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3556.219978] Lustre: DEBUG MARKER: dmesg [ 3556.710939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_101g skipping ALWAYS excluded test 101g [ 3556.891073] Lustre: DEBUG MARKER: SKIP: sanity test_101g skipping ALWAYS excluded test 101g [ 3557.218220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102a: user xattr test ================================================================= 19:41:43 \(1469500903\) [ 3557.431122] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ================================================================= 19:41:43 (1469500903) [ 3557.816747] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3558.263037] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3558.670640] Lustre: DEBUG MARKER: dmesg [ 3559.247142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102b: getfattr\/setfattr for trusted.lov EAs =========================================== 19:41:46 \(1469500906\) [ 3559.430329] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs =========================================== 19:41:46 (1469500906) [ 3559.822202] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3560.281705] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3560.725702] Lustre: DEBUG MARKER: dmesg [ 3561.272309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102c: non-root getfattr\/setfattr for lustre.lov EAs =================================== 19:41:48 \(1469500908\) [ 3561.481990] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs =================================== 19:41:48 (1469500908) [ 3563.495852] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3563.942564] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3564.400415] Lustre: DEBUG MARKER: dmesg [ 3564.926575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 19:41:51 \(1469500911\) [ 3565.161393] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 19:41:51 (1469500911) [ 3565.430496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102d lustre-aware tar is not installed [ 3565.651893] Lustre: DEBUG MARKER: SKIP: sanity test_102d lustre-aware tar is not installed [ 3565.920044] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3566.316939] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3566.739549] Lustre: DEBUG MARKER: dmesg [ 3567.501181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102f: tar copy files, not keep osts =================================================== 19:41:54 \(1469500914\) [ 3567.743392] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts =================================================== 19:41:54 (1469500914) [ 3567.970218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102f lustre-aware tar is not installed [ 3568.162215] Lustre: DEBUG MARKER: SKIP: sanity test_102f lustre-aware tar is not installed [ 3568.439418] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3568.900689] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3569.394750] Lustre: DEBUG MARKER: dmesg [ 3569.938369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102h: grow xattr from inside inode to external block ================================== 19:41:56 \(1469500916\) [ 3570.142565] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ================================== 19:41:56 (1469500916) [ 3570.501088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.big on \/mnt\/lustre\/f102h.sanity [ 3570.702620] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 3571.030302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.sml on \/mnt\/lustre\/f102h.sanity [ 3571.230123] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 3571.534790] Lustre: DEBUG MARKER: /usr/sbin/lctl mark grow trusted.sml on \/mnt\/lustre\/f102h.sanity [ 3571.732105] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 3571.987302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trusted.big still valid after growing trusted.sml [ 3572.186656] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 3572.473809] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3572.921748] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3573.321964] Lustre: DEBUG MARKER: dmesg [ 3573.840889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102ha: grow xattr from inside inode to external inode ================================= 19:42:00 \(1469500920\) [ 3574.016819] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ================================= 19:42:00 (1469500920) [ 3574.349411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.big on \/mnt\/lustre\/f102ha.sanity [ 3574.569634] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 3574.859926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.sml on \/mnt\/lustre\/f102ha.sanity [ 3575.081039] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 3575.357299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark grow trusted.sml on \/mnt\/lustre\/f102ha.sanity [ 3575.568802] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 3575.880472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trusted.big still valid after growing trusted.sml [ 3576.092972] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 3576.339840] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3576.780026] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3577.270920] Lustre: DEBUG MARKER: dmesg [ 3577.850735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102i: lgetxattr test on symbolic link ================================================= 19:42:04 \(1469500924\) [ 3578.083371] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ================================================= 19:42:04 (1469500924) [ 3578.381167] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3578.808651] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3579.221630] Lustre: DEBUG MARKER: dmesg [ 3579.761546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 19:42:06 \(1469500926\) [ 3579.954851] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 19:42:06 (1469500926) [ 3580.232045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102j lustre-aware tar is not installed [ 3580.462370] Lustre: DEBUG MARKER: SKIP: sanity test_102j lustre-aware tar is not installed [ 3580.737982] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3581.135603] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3581.550576] Lustre: DEBUG MARKER: dmesg [ 3582.177738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102k: setfattr without parameter of value shouldn\'t cause a crash ===================== 19:42:08 \(1469500928\) [ 3582.393197] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ===================== 19:42:08 (1469500928) [ 3582.754470] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3583.232971] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3583.750814] Lustre: DEBUG MARKER: dmesg [ 3584.289519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102l: listxattr size test ============================================================= 19:42:11 \(1469500931\) [ 3584.492112] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================= 19:42:11 (1469500931) [ 3584.777606] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3585.189661] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3585.629550] Lustre: DEBUG MARKER: dmesg [ 3586.170212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 19:42:12 \(1469500932\) [ 3586.363323] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 19:42:12 (1469500932) [ 3586.652248] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3587.063486] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3587.497906] Lustre: DEBUG MARKER: dmesg [ 3588.035347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 19:42:14 \(1469500934\) [ 3588.223116] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 19:42:14 (1469500934) [ 3588.412918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3588.847884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3589.257213] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3589.667842] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3590.042791] Lustre: DEBUG MARKER: dmesg [ 3590.595107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102p: check setxattr\(2\) correctly fails without permission ============================ 19:42:17 \(1469500937\) [ 3590.786150] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ============================ 19:42:17 (1469500937) [ 3590.969767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3591.516315] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3591.940947] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3592.406976] Lustre: DEBUG MARKER: dmesg [ 3593.078777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 19:42:19 \(1469500939\) [ 3593.299465] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 19:42:19 (1469500939) [ 3593.488175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3593.927017] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3594.379761] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3594.810776] Lustre: DEBUG MARKER: dmesg [ 3595.304603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102r: set EAs with empty values ======================================================= 19:42:22 \(1469500942\) [ 3595.520387] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values ======================================================= 19:42:22 (1469500942) [ 3595.689916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3596.261367] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3596.705874] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3597.170385] Lustre: DEBUG MARKER: dmesg [ 3597.721225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103a: acl test ======================================================================== 19:42:24 \(1469500944\) [ 3597.956555] Lustre: DEBUG MARKER: == sanity test 103a: acl test ======================================================================== 19:42:24 (1469500944) [ 3598.400928] Lustre: DEBUG MARKER: gpasswd -a daemon bin [ 3599.749984] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*0 [ 3600.050717] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.identity_upcall [ 3600.352186] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_upcall /usr/sbin/l_getidentity [ 3600.660800] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0000/identity_flush=-1 [ 3666.573758] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3666.959404] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3667.475117] Lustre: DEBUG MARKER: dmesg [ 3668.036649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103b: MDS mount option \'noacl\' ======================================================== 19:43:34 \(1469501014\) [ 3668.230695] Lustre: DEBUG MARKER: == sanity test 103b: MDS mount option 'noacl' ======================================================== 19:43:34 (1469501014) [ 3668.409021] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3668.701605] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3668.855903] Lustre: Failing over lustre-MDT0000 [ 3679.513298] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.87@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.668565] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.86@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.374055] Lustre: 27147:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469501022/real 1469501022] req@ffff880049dff000 x1540879853793680/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469501028 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3683.044076] Lustre: server umount lustre-MDT0000 complete [ 3683.191730] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3683.500762] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 3697.755714] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 3698.038592] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 3701.209036] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 3701.518088] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o noacl lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 3701.715281] Lustre: MGS: Connection restored to f14df7fe-c07f-3b6c-2e32-0b0fa5cf5034 (at 0@lo) [ 3701.719023] Lustre: Skipped 7 previous similar messages [ 3702.851148] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.85@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3702.859701] LustreError: Skipped 2 previous similar messages [ 3705.009117] LustreError: 28613:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 3711.810930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3712.128657] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3712.678885] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3712.787291] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3713.120036] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3713.409773] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3713.728628] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 3714.030058] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3724.528850] Lustre: lustre-MDT0000: Recovery over after 0:12, of 2 clients 2 recovered and 0 were evicted. [ 3724.764600] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3725.082411] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3725.240474] Lustre: Failing over lustre-MDT0000 [ 3727.564169] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.87@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.570657] LustreError: Skipped 1 previous similar message [ 3732.679991] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.86@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.690833] LustreError: Skipped 2 previous similar messages [ 3733.384121] Lustre: 29364:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469501073/real 1469501073] req@ffff880055838f00 x1540879853810752/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469501079 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3734.165248] Lustre: server umount lustre-MDT0000 complete [ 3734.319799] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3734.641461] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 3737.350616] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 3737.689692] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 3740.201676] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 3740.520959] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 3742.387536] LustreError: 30827:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 3746.268668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3746.570870] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3747.133196] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3747.429539] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3747.748411] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3748.039875] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 3748.338105] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3748.719843] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3749.012562] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3749.192101] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3749.666775] Lustre: DEBUG MARKER: dmesg [ 3750.255765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103c: \'cp -rp\' won\'t set empty acl ==================================================== 19:44:57 \(1469501097\) [ 3750.454593] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ==================================================== 19:44:57 (1469501097) [ 3755.682564] Lustre: lustre-MDT0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 3758.144036] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3758.598112] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3759.078328] Lustre: DEBUG MARKER: dmesg [ 3759.544318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 104a: lfs df [-ih] [path] test ======================================================== 19:45:06 \(1469501106\) [ 3759.778467] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test ======================================================== 19:45:06 (1469501106) [ 3760.250167] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff8800554c1800.ost_server_uuid in FULL state after 0 sec [ 3760.414915] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800554c1800.ost_server_uuid in FULL state after 0 sec [ 3760.708602] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3761.145170] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3761.655302] Lustre: DEBUG MARKER: dmesg [ 3762.243908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 19:45:08 \(1469501108\) [ 3762.578554] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 19:45:08 (1469501108) [ 3762.876477] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3763.387626] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3763.808230] Lustre: DEBUG MARKER: dmesg [ 3764.399610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105a: flock when mounted without -o flock test ======================================== 19:45:11 \(1469501111\) [ 3764.619095] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ======================================== 19:45:11 (1469501111) [ 3764.912460] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3765.345583] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3765.844945] Lustre: DEBUG MARKER: dmesg [ 3766.410728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105b: fcntl when mounted without -o flock test ======================================== 19:45:13 \(1469501113\) [ 3766.644631] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ======================================== 19:45:13 (1469501113) [ 3766.939136] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3767.394431] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3767.880450] Lustre: DEBUG MARKER: dmesg [ 3768.458939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105c: lockf when mounted without -o flock test ======================================== 19:45:15 \(1469501115\) [ 3768.641065] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ======================================== 19:45:15 (1469501115) [ 3768.909437] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3769.358124] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3769.798051] Lustre: DEBUG MARKER: dmesg [ 3770.289634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105d: flock race \(should not freeze\) ================================================== 19:45:17 \(1469501117\) [ 3770.494318] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================== 19:45:17 (1469501117) [ 3780.810517] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3781.215641] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3781.671066] Lustre: DEBUG MARKER: dmesg [ 3782.244865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105e: Two conflicting flocks from same process ======================================== 19:45:28 \(1469501128\) [ 3782.422601] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ======================================== 19:45:28 (1469501128) [ 3782.719204] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3783.127848] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3783.545697] Lustre: DEBUG MARKER: dmesg [ 3784.090393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 19:45:30 \(1469501130\) [ 3784.287872] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 19:45:30 (1469501130) [ 3784.591700] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3784.989548] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3785.446836] Lustre: DEBUG MARKER: dmesg [ 3785.988827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 107: Coredump on SIG ================================================================== 19:45:32 \(1469501132\) [ 3786.182975] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ================================================================== 19:45:32 (1469501132) [ 3787.519910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3787.989704] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3788.400639] Lustre: DEBUG MARKER: dmesg [ 3788.974043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 110: filename length checking ========================================================= 19:45:35 \(1469501135\) [ 3789.170034] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ========================================================= 19:45:35 (1469501135) [ 3789.425823] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3789.854960] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3790.254027] Lustre: DEBUG MARKER: dmesg [ 3790.742940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_115 skipping SLOW test 115 [ 3790.980587] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 3791.309449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 116a: stripe QOS: free space balance ================================================== 19:45:38 \(1469501138\) [ 3791.511247] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ================================================== 19:45:38 (1469501138) [ 3791.689085] Lustre: DEBUG MARKER: lctl get_param -n lo*.*-mdtlov.qos_prio_free [ 3792.032654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3792.389206] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3803.673853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3805.050424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3806.392322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3812.730576] Lustre: DEBUG MARKER: lctl get_param -n *.*MDT0000-mdtlov.qos_threshold_rr [ 3879.960621] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 3890.295608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3900.650635] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 3910.991844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3915.050969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3915.448723] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3924.114938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3925.510531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3926.877253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3928.242263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3929.662243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3931.026930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3932.408742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3938.934159] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3939.395103] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3939.811080] Lustre: DEBUG MARKER: dmesg [ 3940.489574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 116b: QoS shouldn\'t LBUG if not enough OSTs found on the 2nd pass ===================== 19:48:07 \(1469501287\) [ 3940.695244] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ===================== 19:48:07 (1469501287) [ 3940.881152] Lustre: DEBUG MARKER: lctl get_param -n lo*.lustre-MDT0000-mdtlov.qos_threshold_rr [ 3941.191369] Lustre: DEBUG MARKER: lctl set_param lo*.lustre-MDT0000-mdtlov.qos_threshold_rr=0 [ 3941.509431] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x147 [ 3941.669728] Lustre: *** cfs_fail_loc=147, val=0*** [ 3941.869855] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3942.251568] Lustre: DEBUG MARKER: lctl set_param lo*.lustre-MDT0000-mdtlov.qos_threshold_rr=17% [ 3942.633902] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3943.133474] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3943.569962] Lustre: DEBUG MARKER: dmesg [ 3944.088075] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 117: verify osd extend ================================================================ 19:48:10 \(1469501290\) [ 3944.305301] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ================================================================ 19:48:10 (1469501290) [ 3944.562970] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3944.945154] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3945.448310] Lustre: DEBUG MARKER: dmesg [ 3945.999806] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118a: verify O_SYNC works ============================================================= 19:48:12 \(1469501292\) [ 3946.231191] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============================================================= 19:48:12 (1469501292) [ 3951.833607] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3952.291767] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3952.762380] Lustre: DEBUG MARKER: dmesg [ 3953.314467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 19:48:20 \(1469501300\) [ 3953.528754] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 19:48:20 (1469501300) [ 3972.554136] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3972.983098] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3973.458203] Lustre: DEBUG MARKER: dmesg [ 3974.031040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 19:48:40 \(1469501320\) [ 3974.256695] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 19:48:40 (1469501320) [ 3980.583081] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3980.994664] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3981.402144] Lustre: DEBUG MARKER: dmesg [ 3981.951529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 19:48:48 \(1469501328\) [ 3982.204481] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 19:48:48 (1469501328) [ 3989.327841] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3989.726610] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3990.136834] Lustre: DEBUG MARKER: dmesg [ 3990.731239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 19:48:57 \(1469501337\) [ 3990.923717] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 19:48:57 (1469501337) [ 3995.215528] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3995.670372] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 3996.182125] Lustre: DEBUG MARKER: dmesg [ 3996.700375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118g: Don\'t stay in wait if we got local -ENOMEM ===================================== 19:49:03 \(1469501343\) [ 3996.885312] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ===================================== 19:49:03 (1469501343) [ 3999.953034] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4000.374704] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4000.789094] Lustre: DEBUG MARKER: dmesg [ 4001.371928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118h: Verify timeout in handling recoverables errors ================================= 19:49:08 \(1469501348\) [ 4001.560883] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ================================= 19:49:08 (1469501348) [ 4014.558641] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4015.122922] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4015.521344] Lustre: DEBUG MARKER: dmesg [ 4016.097161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118i: Fix error before timeout in recoverable error ================================== 19:49:22 \(1469501362\) [ 4016.263986] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ================================== 19:49:22 (1469501362) [ 4030.163086] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4030.555519] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4030.994829] Lustre: DEBUG MARKER: dmesg [ 4031.535115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118j: Simulate unrecoverable OST side error =========================================== 19:49:38 \(1469501378\) [ 4031.745739] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error =========================================== 19:49:38 (1469501378) [ 4036.618388] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4037.036290] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4037.491209] Lustre: DEBUG MARKER: dmesg [ 4038.013422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 19:49:44 \(1469501384\) [ 4038.281602] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 19:49:44 (1469501384) [ 4044.704021] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4045.731751] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4046.211489] Lustre: DEBUG MARKER: dmesg [ 4049.944093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118l: fsync dir ======================================================================= 19:49:56 \(1469501396\) [ 4050.133580] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir ======================================================================= 19:49:56 (1469501396) [ 4063.967729] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4064.380886] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4064.830583] Lustre: DEBUG MARKER: dmesg [ 4065.382148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118m: fdatasync dir =================================================================== 19:50:12 \(1469501412\) [ 4065.565036] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =================================================================== 19:50:12 (1469501412) [ 4068.310845] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4068.733605] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4069.169968] Lustre: DEBUG MARKER: dmesg [ 4069.870610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119a: Short directIO read must return actual read amount ============================== 19:50:16 \(1469501416\) [ 4070.079100] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ============================== 19:50:16 (1469501416) [ 4070.730502] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4071.142862] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4071.565164] Lustre: DEBUG MARKER: dmesg [ 4072.039607] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119b: Sparse directIO read must return actual read amount ============================= 19:50:18 \(1469501418\) [ 4072.224495] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ============================= 19:50:18 (1469501418) [ 4072.970999] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4073.380403] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4073.869464] Lustre: DEBUG MARKER: dmesg [ 4074.331393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119c: Testing for direct read hitting hole ============================================ 19:50:21 \(1469501421\) [ 4074.550962] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ============================================ 19:50:21 (1469501421) [ 4075.335455] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4075.751594] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4076.169223] Lustre: DEBUG MARKER: dmesg [ 4076.678902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 19:50:23 \(1469501423\) [ 4076.922674] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 19:50:23 (1469501423) [ 4079.393381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark the DIO writes have completed, now wait for the reads \(should not block very long\) [ 4079.592800] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 4081.930443] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4082.335423] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4082.748489] Lustre: DEBUG MARKER: dmesg [ 4083.317847] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 19:50:30 \(1469501430\) [ 4083.540828] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 19:50:30 (1469501430) [ 4083.790866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4084.150020] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4084.529526] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4085.019606] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4085.471966] Lustre: DEBUG MARKER: dmesg [ 4086.036100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120b: Early Lock Cancel: create test ================================================== 19:50:32 \(1469501432\) [ 4086.223533] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ================================================== 19:50:32 (1469501432) [ 4086.479913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4086.806121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4087.199579] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4087.650900] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4088.173941] Lustre: DEBUG MARKER: dmesg [ 4088.826689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120c: Early Lock Cancel: link test ==================================================== 19:50:35 \(1469501435\) [ 4089.033684] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ==================================================== 19:50:35 (1469501435) [ 4089.319909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4089.641155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4090.068753] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4090.481848] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4090.946357] Lustre: DEBUG MARKER: dmesg [ 4091.484889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120d: Early Lock Cancel: setattr test ================================================= 19:50:38 \(1469501438\) [ 4091.734698] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ================================================= 19:50:38 (1469501438) [ 4091.978445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4092.290347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4092.651763] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4093.071148] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4093.514664] Lustre: DEBUG MARKER: dmesg [ 4094.024285] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120e: Early Lock Cancel: unlink test ================================================== 19:50:40 \(1469501440\) [ 4094.196321] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ================================================== 19:50:40 (1469501440) [ 4094.412271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4099.727027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4100.083702] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4100.484250] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4100.869351] Lustre: DEBUG MARKER: dmesg [ 4101.422427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120f: Early Lock Cancel: rename test ================================================== 19:50:48 \(1469501448\) [ 4101.632333] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ================================================== 19:50:48 (1469501448) [ 4101.871392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4107.189445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4107.558117] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4108.041764] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4108.466069] Lustre: DEBUG MARKER: dmesg [ 4109.011925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120g: Early Lock Cancel: performance test ============================================= 19:50:55 \(1469501455\) [ 4109.193896] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ============================================= 19:50:55 (1469501455) [ 4109.413406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4136.113071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4154.600248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4157.030571] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4157.462220] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4157.872372] Lustre: DEBUG MARKER: dmesg [ 4158.357467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 121: read cancel race ================================================================= 19:51:45 \(1469501505\) [ 4158.593135] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ================================================================= 19:51:45 (1469501505) [ 4158.882054] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4159.380471] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4159.784715] Lustre: DEBUG MARKER: dmesg [ 4160.344571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 123a: verify statahead work =========================================================== 19:51:47 \(1469501507\) [ 4160.604475] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work =========================================================== 19:51:47 (1469501507) [ 4161.419880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 100 files without statahead: 1 sec [ 4161.610720] Lustre: DEBUG MARKER: ls 100 files without statahead: 1 sec [ 4162.020590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 100 files with statahead: 0 sec [ 4162.211229] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [ 4167.349830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 1000 files without statahead: 3 sec [ 4167.508351] Lustre: DEBUG MARKER: ls 1000 files without statahead: 3 sec [ 4168.249119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 1000 files with statahead: 0 sec [ 4168.418905] Lustre: DEBUG MARKER: ls 1000 files with statahead: 0 sec [ 4221.690463] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 10000 files without statahead: 28 sec [ 4221.902866] Lustre: DEBUG MARKER: ls 10000 files without statahead: 28 sec [ 4225.874662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 10000 files with statahead: 3 sec [ 4226.062495] Lustre: DEBUG MARKER: ls 10000 files with statahead: 3 sec [ 4226.330813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls done [ 4226.512912] Lustre: DEBUG MARKER: ls done [ 4245.344791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark rm -r \/mnt\/lustre\/d123a.sanity\/: 19 seconds [ 4245.552005] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 19 seconds [ 4245.788752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark rm done [ 4246.002374] Lustre: DEBUG MARKER: rm done [ 4246.200698] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4246.666303] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4247.166718] Lustre: DEBUG MARKER: dmesg [ 4247.792258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 123b: not panic with network error in statahead enqueue \(bug 15027\) =================== 19:53:14 \(1469501594\) [ 4247.997549] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) =================== 19:53:14 (1469501594) [ 4254.496392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls done [ 4254.683103] Lustre: DEBUG MARKER: ls done [ 4256.889635] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4257.364184] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4257.864513] Lustre: DEBUG MARKER: dmesg [ 4258.448651] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124a: lru resize ====================================================================== 19:53:25 \(1469501605\) [ 4258.677483] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ====================================================================== 19:53:25 (1469501605) [ 4258.957585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create 2000 files at \/mnt\/lustre\/d124a.sanity [ 4259.244864] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 4266.352070] Lustre: DEBUG MARKER: /usr/sbin/lctl mark NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800554c1800 [ 4266.563578] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800554c1800 [ 4266.833766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800554c1800 [ 4267.022906] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800554c1800 [ 4267.289854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LRU=2003 [ 4267.481308] Lustre: DEBUG MARKER: LRU=2003 [ 4267.728626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LIMIT=91949 [ 4267.921893] Lustre: DEBUG MARKER: LIMIT=91949 [ 4268.153596] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LVF=55086 [ 4268.371202] Lustre: DEBUG MARKER: LVF=55086 [ 4268.649835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OLD_LVF=1 [ 4268.883869] Lustre: DEBUG MARKER: OLD_LVF=1 [ 4269.111255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Sleep 50 sec [ 4269.317211] Lustre: DEBUG MARKER: Sleep 50 sec [ 4319.705708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Dropped 644 locks in 50s [ 4319.891830] Lustre: DEBUG MARKER: Dropped 644 locks in 50s [ 4320.134422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark unlink 2000 files at \/mnt\/lustre\/d124a.sanity [ 4320.361530] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 4324.658120] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4325.100767] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4325.569062] Lustre: DEBUG MARKER: dmesg [ 4326.098392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124b: lru resize \(performance test\) =================================================== 19:54:32 \(1469501672\) [ 4326.349367] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) =================================================== 19:54:32 (1469501672) [ 4334.578495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d124b.sanity\/disable_lru_resize 3 times [ 4334.814277] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 4339.923324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls -la time: 5 seconds [ 4340.130383] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 4340.388807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lru_size = 200 [ 4340.603023] Lustre: DEBUG MARKER: lru_size = 200 [ 4361.574638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d124b.sanity\/enable_lru_resize 3 times [ 4361.744506] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 4366.475785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls -la time: 5 seconds [ 4366.661156] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 4366.970583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lru_size = 4006 [ 4367.145555] Lustre: DEBUG MARKER: lru_size = 4006 [ 4367.412107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lru resize performs the same with no lru resize [ 4367.616299] Lustre: DEBUG MARKER: lru resize performs the same with no lru resize [ 4375.993631] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4376.425856] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4376.856606] Lustre: DEBUG MARKER: dmesg [ 4377.342590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124c: LRUR cancel very aged locks ===================================================== 19:55:24 \(1469501724\) [ 4377.572966] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ===================================================== 19:55:24 (1469501724) [ 4398.436554] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4398.912751] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4399.389114] Lustre: DEBUG MARKER: dmesg [ 4400.056758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 125: don\'t return EPROTO when a dir has a non-default striping and ACLs =============== 19:55:46 \(1469501746\) [ 4400.280307] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs =============== 19:55:46 (1469501746) [ 4403.482444] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4403.953537] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4404.409642] Lustre: DEBUG MARKER: dmesg [ 4404.950052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 126: check that the fsgid provided by the client is taken into account ================ 19:55:51 \(1469501751\) [ 4405.153223] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ================ 19:55:51 (1469501751) [ 4405.441218] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4405.863865] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4406.370521] Lustre: DEBUG MARKER: dmesg [ 4406.927329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 127a: verify the client stats are sane ================================================ 19:55:53 \(1469501753\) [ 4407.131686] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ================================================ 19:55:53 (1469501753) [ 4407.596901] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4408.022615] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4408.478960] Lustre: DEBUG MARKER: dmesg [ 4409.125644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 127b: verify the llite client stats are sane ========================================== 19:55:55 \(1469501755\) [ 4409.360570] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================== 19:55:55 (1469501755) [ 4409.724568] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4410.108575] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4410.572728] Lustre: DEBUG MARKER: dmesg [ 4411.070532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 128: interactive lfs for 2 consecutive find\'s ========================================= 19:55:57 \(1469501757\) [ 4411.252625] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================= 19:55:57 (1469501757) [ 4411.593976] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4411.989926] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4412.433067] Lustre: DEBUG MARKER: dmesg [ 4412.982115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 129: test directory size limit ======================================================== 19:55:59 \(1469501759\) [ 4413.255222] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ======================================================== 19:55:59 (1469501759) [ 4413.535615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_129 Only applicable to ldiskfs-based MDTs [ 4413.746591] Lustre: DEBUG MARKER: SKIP: sanity test_129 Only applicable to ldiskfs-based MDTs [ 4413.995643] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4414.425594] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4414.903932] Lustre: DEBUG MARKER: dmesg [ 4415.490223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130a: FIEMAP \(1-stripe file\) ========================================================== 19:56:02 \(1469501762\) [ 4415.687595] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ========================================================== 19:56:02 (1469501762) [ 4420.376437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130a ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 4420.564610] Lustre: DEBUG MARKER: SKIP: sanity test_130a ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 4420.836955] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4421.345146] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4421.835175] Lustre: DEBUG MARKER: dmesg [ 4422.392983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130b: FIEMAP \(2-stripe file\) ========================================================== 19:56:09 \(1469501769\) [ 4422.614141] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ========================================================== 19:56:09 (1469501769) [ 4422.937113] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130b ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 4423.126514] Lustre: DEBUG MARKER: SKIP: sanity test_130b ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 4423.385543] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4423.816115] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4424.282482] Lustre: DEBUG MARKER: dmesg [ 4424.871577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130c: FIEMAP \(2-stripe file with hole\) ================================================ 19:56:11 \(1469501771\) [ 4425.044556] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ================================================ 19:56:11 (1469501771) [ 4425.402636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130c ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 4425.590846] Lustre: DEBUG MARKER: SKIP: sanity test_130c ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 4425.853083] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4426.308779] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4426.734570] Lustre: DEBUG MARKER: dmesg [ 4427.348581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130d: FIEMAP \(N-stripe file\) ========================================================== 19:56:14 \(1469501774\) [ 4427.543855] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ========================================================== 19:56:14 (1469501774) [ 4427.850317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130d skipping FIEMAP on N-stripe file test [ 4428.078420] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping FIEMAP on N-stripe file test [ 4428.364611] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4428.788709] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4429.203741] Lustre: DEBUG MARKER: dmesg [ 4429.860574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130e: FIEMAP \(test continuation FIEMAP calls\) ========================================= 19:56:16 \(1469501776\) [ 4430.047029] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 19:56:16 (1469501776) [ 4430.381713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130e ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 4430.581537] Lustre: DEBUG MARKER: SKIP: sanity test_130e ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 4430.795974] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4431.275362] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4431.724071] Lustre: DEBUG MARKER: dmesg [ 4432.200524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131a: test iov\'s crossing stripe boundary for writev\/readv ============================ 19:56:19 \(1469501779\) [ 4432.442653] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ============================ 19:56:19 (1469501779) [ 4432.706918] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4433.102317] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4433.499666] Lustre: DEBUG MARKER: dmesg [ 4434.092971] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131b: test append writev ============================================================== 19:56:20 \(1469501780\) [ 4434.265721] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ============================================================== 19:56:20 (1469501780) [ 4434.562107] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4434.978710] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4435.479111] Lustre: DEBUG MARKER: dmesg [ 4436.051873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131c: test read\/write on file w\/o objects ============================================= 19:56:22 \(1469501782\) [ 4436.242607] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ============================================= 19:56:22 (1469501782) [ 4436.561726] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4436.991455] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4437.518721] Lustre: DEBUG MARKER: dmesg [ 4438.036611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131d: test short read ================================================================= 19:56:24 \(1469501784\) [ 4438.223968] Lustre: DEBUG MARKER: == sanity test 131d: test short read ================================================================= 19:56:24 (1469501784) [ 4438.466543] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4438.952287] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4439.389096] Lustre: DEBUG MARKER: dmesg [ 4440.000126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131e: test read hitting hole ========================================================== 19:56:26 \(1469501786\) [ 4440.213894] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 19:56:26 (1469501786) [ 4440.488210] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4440.961343] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4441.375973] Lustre: DEBUG MARKER: dmesg [ 4441.976922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133a: Verifying MDT stats ============================================================= 19:56:28 \(1469501788\) [ 4442.183741] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ============================================================= 19:56:28 (1469501788) [ 4442.422628] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param mdt.*.rename_stats [ 4442.739668] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 4443.400648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4443.732641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4444.042033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4444.351624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4444.670516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4445.009035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4445.343437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4445.670547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4446.002293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4446.401821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4446.948814] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4447.386164] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4447.898256] Lustre: DEBUG MARKER: dmesg [ 4448.490793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133b: Verifying extra MDT stats ======================================================= 19:56:35 \(1469501795\) [ 4448.712748] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ======================================================= 19:56:35 (1469501795) [ 4448.999809] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 4449.658648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4449.968578] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 4450.273771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4450.629517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4450.940442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4451.374742] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4451.861831] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4452.321304] Lustre: DEBUG MARKER: dmesg [ 4452.871171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133c: Verifying OST stats ============================================================= 19:56:39 \(1469501799\) [ 4453.054712] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ============================================================= 19:56:39 (1469501799) [ 4453.679636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4453.979732] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4456.530080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4457.880033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4459.220375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4460.659520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4462.061709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4463.417897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4469.791549] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 4471.609640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4471.927814] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4475.950918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4477.321083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4478.717750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4480.050754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4481.418191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4482.814780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4489.705735] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4490.147804] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4490.555253] Lustre: DEBUG MARKER: dmesg [ 4491.117613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133d: Verifying rename_stats ========================================================== 19:57:17 \(1469501837\) [ 4491.315901] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================================== 19:57:17 (1469501837) [ 4491.600731] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param mdt.*.rename_stats [ 4491.895796] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.rename_stats=clear [ 4493.847872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4494.169954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4494.490181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4494.780923] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.rename_stats=clear [ 4495.110669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4495.410460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4495.760067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4496.081213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4497.696163] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4498.146526] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4498.620724] Lustre: DEBUG MARKER: dmesg [ 4499.182029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133e: Verifying OST {read,write}_bytes nid stats ====================================== 19:57:25 \(1469501845\) [ 4499.370686] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats ====================================== 19:57:25 (1469501845) [ 4501.544899] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4501.985136] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4502.412512] Lustre: DEBUG MARKER: dmesg [ 4503.004182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133f: Check for LBUGs\/Oopses\/unreadable files in \/proc ================================ 19:57:29 \(1469501849\) [ 4503.200124] Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc ================================ 19:57:29 (1469501849) [ 4504.067863] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ ! -name req_history -exec cat {} \; [ 4505.469635] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ ! -name req_history -type f -exec cat {} \; [ 4508.602408] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4509.062245] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4509.480934] Lustre: DEBUG MARKER: dmesg [ 4510.040811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133g: Check for Oopses on bad io area writes\/reads in \/proc =========================== 19:57:36 \(1469501856\) [ 4510.227585] Lustre: DEBUG MARKER: == sanity test 133g: Check for Oopses on bad io area writes/reads in /proc =========================== 19:57:36 (1469501856) [ 4510.988736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4511.691510] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ -type f -not -name force_lbug -not -name changelog_mask -exec badarea_io {} \; [ 4511.853280] LustreError: 28764:0:(ldlm_resource.c:109:seq_watermark_write()) Failed to set lock_limit_mb, rc = -14. [ 4511.893732] LustreError: 28796:0:(ldlm_pool.c:722:lprocfs_wr_recalc_period()) Can't parse user input, rc = -14 [ 4511.905903] LustreError: 28806:0:(ldlm_resource.c:348:lprocfs_lru_size_seq_write()) invalid value written [ 4512.077580] Lustre: 29036:0:(mdt_coordinator.c:1808:mdt_hsm_policy_seq_write()) lustre-MDT0000: '\x03' is unknown, supported policies are: [ 4512.088797] LustreError: 29048:0:(mdt_coordinator.c:1960:mdt_hsm_cdt_control_seq_write()) lustre-MDT0000: Valid coordinator control commands are: enabled shutdown disabled purge help [ 4512.096583] Lustre: 29054:0:(lprocfs_status.c:2209:lprocfs_wr_nosquash_nids()) lustre-MDT0000: failed to set nosquash_nids to "", bad address rc = -14 [ 4512.100624] Lustre: 29054:0:(lprocfs_status.c:2213:lprocfs_wr_nosquash_nids()) lustre-MDT0000: failed to set nosquash_nids due to string too long rc = -22 [ 4512.105687] Lustre: 29055:0:(lprocfs_status.c:2141:lprocfs_wr_root_squash()) lustre-MDT0000: failed to set root_squash due to bad address, rc = -14 [ 4512.109921] Lustre: 29055:0:(lprocfs_status.c:2137:lprocfs_wr_root_squash()) lustre-MDT0000: failed to set root_squash to "", needs uid:gid format, rc = -22 [ 4512.116858] LustreError: 29059:0:(genops.c:1519:obd_export_evict_by_uuid()) lustre-MDT0000: can't disconnect : no exports found [ 4512.122494] LustreError: 29061:0:(mdt_lproc.c:370:lprocfs_identity_info_seq_write()) lustre-MDT0000: invalid data count = 5, size = 1048 [ 4512.126848] LustreError: 29061:0:(mdt_lproc.c:387:lprocfs_identity_info_seq_write()) lustre-MDT0000: MDS identity downcall bad params [ 4512.133107] LustreError: 29063:0:(mdt_lproc.c:302:mdt_identity_upcall_seq_write()) lustre-MDT0000: identity upcall too long [ 4512.262979] LustreError: 29190:0:(mgs_nids.c:793:lprocfs_wr_ir_state()) Unable to process command: \x03(-22) [ 4514.196935] LustreError: 29258:0:(lproc_fid.c:180:lprocfs_server_fid_width_seq_write()) srv-lustre-MDT0000: invalid FID sequence width: rc = -14 [ 4514.239159] LustreError: 29276:0:(nodemap_handler.c:1037:nodemap_create()) cannot add nodemap: '\x03': rc = -22 [ 4514.265240] Lustre: 29293:0:(libcfs_string.c:131:cfs_str2mask()) unknown mask '\x03'. mask usage: [+|-] ... [ 4514.272751] Lustre: 0 MB is too small for debug buffer size, setting it to 2 MB. [ 4515.560717] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4515.869604] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4516.013386] Lustre: Failing over lustre-MDT0000 [ 4523.768391] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.86@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4527.560462] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.87@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4527.676142] Lustre: 29472:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469501867/real 1469501867] req@ffff880049342d00 x1540879854366288/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469501873 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4530.008743] Lustre: server umount lustre-MDT0000 complete [ 4530.179234] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4530.531804] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 4544.421986] Lustre: DEBUG MARKER: hostname [ 4544.950183] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 4548.613584] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 4548.930285] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 4549.119439] Lustre: MGS: Connection restored to f14df7fe-c07f-3b6c-2e32-0b0fa5cf5034 (at 0@lo) [ 4549.123553] Lustre: Skipped 17 previous similar messages [ 4552.862039] LustreError: 30963:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4557.540247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4557.858768] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4558.459805] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4558.779904] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4559.070707] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4559.369398] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 4559.577350] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4564.905871] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4565.464678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4565.466035] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4565.635714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4565.640149] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4568.058531] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4568.378543] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 4569.936708] Lustre: lustre-MDT0000: Not available for connect from 10.2.5.85@tcp (stopping) [ 4569.940742] Lustre: Skipped 1 previous similar message [ 4569.951110] LustreError: 7955:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880053b79800 x1540879854367120/t0(0) o13->lustre-OST0000-osc-MDT0000@10.2.5.85@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 4569.960854] LustreError: 7955:0:(client.c:1168:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 4579.912699] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.85@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.918603] LustreError: Skipped 3 previous similar messages [ 4580.570057] Lustre: 31816:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469501920/real 1469501920] req@ffff880053b79800 x1540879854367216/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469501926 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4580.592168] Lustre: server umount lustre-MDT0000 complete [ 4580.739103] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4581.054748] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 4611.580957] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 4612.001715] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre-mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi [ 4613.280789] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 4613.629503] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 4615.129133] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 4615.429820] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 4616.399987] LustreError: 1146:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4617.717208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4618.080746] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4618.757071] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4619.079643] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4619.379692] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4619.698904] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 4620.010428] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4621.406166] Lustre: 7953:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469501962/real 1469501962] req@ffff88003d60cc00 x1540879854367376/t0(0) o8->lustre-OST0001-osc-MDT0000@10.2.5.85@tcp:28/4 lens 520/544 e 0 to 1 dl 1469501967 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4635.627685] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 4636.117636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 4636.280464] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4636.541331] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 4642.046977] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4642.484969] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4642.971906] Lustre: DEBUG MARKER: dmesg [ 4644.764538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 19:59:51 \(1469501991\) [ 4645.001068] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 19:59:51 (1469501991) [ 4645.181097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4649.118916] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x327 [ 4649.438279] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=500 [ 4649.597442] Lustre: *** cfs_fail_loc=327, val=0*** [ 4659.849851] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 4660.145367] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 [ 4663.757524] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4664.196878] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4664.649543] Lustre: DEBUG MARKER: dmesg [ 4665.205937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 20:00:11 \(1469502011\) [ 4665.401687] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 20:00:11 (1469502011) [ 4665.559785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4665.919170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.lock_reclaim_threshold_mb [ 4666.219462] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.lock_reclaim_threshold_mb=0 [ 4666.520055] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x328 [ 4666.838173] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=500 [ 4666.987155] Lustre: *** cfs_fail_loc=328, val=0*** [ 4666.988825] Lustre: Skipped 512 previous similar messages [ 4667.991668] Lustre: *** cfs_fail_loc=328, val=0*** [ 4667.995902] Lustre: Skipped 277 previous similar messages [ 4674.852963] Lustre: *** cfs_fail_loc=328, val=0*** [ 4674.857331] Lustre: Skipped 223 previous similar messages [ 4679.714441] Lustre: *** cfs_fail_loc=328, val=0*** [ 4687.160837] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 4687.489275] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 [ 4687.800269] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.lock_reclaim_threshold_mb=368m [ 4696.068522] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4696.512084] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4697.007338] Lustre: DEBUG MARKER: dmesg [ 4697.514248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 140: Check reasonable stack depth \(shouldn\'t LBUG\) ==================================== 20:00:44 \(1469502044\) [ 4697.748178] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 20:00:44 (1469502044) [ 4701.164416] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4701.620973] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4702.053874] Lustre: DEBUG MARKER: dmesg [ 4702.539120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150: truncate\/append tests ============================================================ 20:00:49 \(1469502049\) [ 4702.800338] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 20:00:49 (1469502049) [ 4704.092639] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4704.509971] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4704.973906] Lustre: DEBUG MARKER: dmesg [ 4705.525336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 151: test cache on oss and controls =================================================== 20:00:52 \(1469502052\) [ 4705.693566] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls =================================================== 20:00:52 (1469502052) [ 4706.458834] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4706.927101] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4707.427676] Lustre: DEBUG MARKER: dmesg [ 4708.014436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 152: test read\/write with enomem ====================================================== 20:00:54 \(1469502054\) [ 4708.235833] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================== 20:00:54 (1469502054) [ 4709.051189] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4709.469985] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4709.944775] Lustre: DEBUG MARKER: dmesg [ 4710.443207] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 153: test if fdatasync does not crash ================================================= 20:00:57 \(1469502057\) [ 4710.628187] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================= 20:00:57 (1469502057) [ 4713.252124] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4713.666851] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4714.145323] Lustre: DEBUG MARKER: dmesg [ 4714.678798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 20:01:01 \(1469502061\) [ 4714.884270] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 20:01:01 (1469502061) [ 4715.059407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4715.477631] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4715.885718] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4716.394376] Lustre: DEBUG MARKER: dmesg [ 4716.930749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154a: Open-by-FID ===================================================================== 20:01:03 \(1469502063\) [ 4717.220315] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ===================================================================== 20:01:03 (1469502063) [ 4717.408468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4717.777213] LustreError: 1128:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 4717.857958] Lustre: 1128:0:(mdd_device.c:584:obf_lookup()) lustre-MDD0000: Trying to lookup invalid FID [f154a.sanity] in .lustre/fid, FID format should be [0x200000400:0x1:0x0] [ 4717.905179] Lustre: 1128:0:(mdd_device.c:594:obf_lookup()) lustre-MDD0000: Trying to lookup invalid FID [0x1:0x2:0x0] in .lustre/fid, sequence should be >= 0x200000400 or within [0xc,0xffffffff]. [ 4718.117957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4719.730692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4720.233078] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4720.696515] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4721.240615] Lustre: DEBUG MARKER: dmesg [ 4721.983712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154b: Open-by-FID for remote directory ================================================ 20:01:08 \(1469502068\) [ 4722.180823] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ================================================ 20:01:08 (1469502068) [ 4722.380578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4722.777484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_154b needs \>= 2 MDTs [ 4722.948264] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [ 4723.199789] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4723.649717] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4724.165645] Lustre: DEBUG MARKER: dmesg [ 4724.818610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 20:01:11 \(1469502071\) [ 4724.992107] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 20:01:11 (1469502071) [ 4725.168743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4725.661311] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4726.168020] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4726.558181] Lustre: DEBUG MARKER: dmesg [ 4727.220804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154d: Verify open file fid ============================================================ 20:01:13 \(1469502073\) [ 4727.461964] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ============================================================ 20:01:13 (1469502073) [ 4727.629190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4728.027942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.exports.'10\.2\.5\.86@tcp'.open_files [ 4728.442108] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4729.009876] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4729.403108] Lustre: DEBUG MARKER: dmesg [ 4730.035306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154e: .lustre is not returned by readdir ============================================== 20:01:16 \(1469502076\) [ 4730.233173] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir ============================================== 20:01:16 (1469502076) [ 4730.420400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4730.830547] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4731.316058] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4731.714697] Lustre: DEBUG MARKER: dmesg [ 4732.383074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154f: get parent fids by reading link ea ============================================== 20:01:19 \(1469502079\) [ 4732.581613] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea ============================================== 20:01:19 (1469502079) [ 4733.120978] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4733.618633] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4734.091192] Lustre: DEBUG MARKER: dmesg [ 4734.729952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154g: various llapi FID tests ========================================================= 20:01:21 \(1469502081\) [ 4734.917957] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ========================================================= 20:01:21 (1469502081) [ 4735.117459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4819.119575] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4819.610926] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4820.023081] Lustre: DEBUG MARKER: dmesg [ 4820.600973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 20:02:47 \(1469502167\) [ 4820.775148] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 20:02:47 (1469502167) [ 4821.961267] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4822.451866] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4822.869785] Lustre: DEBUG MARKER: dmesg [ 4823.510724] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 20:02:50 \(1469502170\) [ 4823.720972] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 20:02:50 (1469502170) [ 4824.845317] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4825.281608] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4825.681750] Lustre: DEBUG MARKER: dmesg [ 4826.257753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 20:02:53 \(1469502173\) [ 4826.449723] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 20:02:53 (1469502173) [ 4827.631091] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4828.088792] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4828.553779] Lustre: DEBUG MARKER: dmesg [ 4829.168992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 20:02:55 \(1469502175\) [ 4829.353159] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 20:02:55 (1469502175) [ 4830.547638] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4831.076924] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4831.498466] Lustre: DEBUG MARKER: dmesg [ 4832.096795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 20:02:58 \(1469502178\) [ 4832.290908] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 20:02:58 (1469502178) [ 4833.348195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4833.662220] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4835.240722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4836.630624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4838.149646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4839.511017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4840.850063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4842.232814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4851.021334] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4851.507578] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4852.011766] Lustre: DEBUG MARKER: dmesg [ 4852.651686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 20:03:19 \(1469502199\) [ 4852.829956] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 20:03:19 (1469502199) [ 4854.082109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4854.380588] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4855.750600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4857.109101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4858.579760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4859.920662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4861.298663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4862.720663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4871.043553] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4871.499864] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4871.963630] Lustre: DEBUG MARKER: dmesg [ 4872.536129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 20:03:39 \(1469502219\) [ 4872.731165] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 20:03:39 (1469502219) [ 4873.871775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4874.174274] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4874.961420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4876.330975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4877.681074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4879.090045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4880.430126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4881.770700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4890.426060] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4890.883074] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4891.298791] Lustre: DEBUG MARKER: dmesg [ 4891.929614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 20:03:58 \(1469502238\) [ 4892.129691] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 20:03:58 (1469502238) [ 4893.199963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4893.509908] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4894.430873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4895.809415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4897.160166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4898.550829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4899.888912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4901.240385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4902.620314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4911.300885] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4911.778904] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4912.224754] Lustre: DEBUG MARKER: dmesg [ 4912.778145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 156: Verification of tunables ========================================================= 20:04:19 \(1469502259\) [ 4912.993264] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ========================================================= 20:04:19 (1469502259) [ 4913.602313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_156 LU-1956\/LU-2261: stats not implemented on OSD ZFS [ 4913.869566] Lustre: DEBUG MARKER: SKIP: sanity test_156 LU-1956/LU-2261: stats not implemented on OSD ZFS [ 4914.091640] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4914.553481] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4915.040870] Lustre: DEBUG MARKER: dmesg [ 4915.678722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160a: changelog sanity ================================================================ 20:04:22 \(1469502262\) [ 4915.905190] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ================================================================ 20:04:22 (1469502262) [ 4916.132139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4916.480423] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4916.651074] Lustre: lustre-MDD0000: changelog on [ 4916.810620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4917.230022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4917.529558] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-MKDIR [ 4917.888352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4918.202048] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-CLOSE [ 4918.544512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4918.910076] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+MKDIR [ 4919.219922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4919.549774] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+CLOSE [ 4919.998751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4920.319741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4920.638680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4920.961643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4921.307957] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4921.619557] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4921.764687] Lustre: Failing over lustre-MDT0000 [ 4928.193955] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.5.86@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.904074] Lustre: 14924:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1469502269/real 1469502269] req@ffff880048d25b00 x1540879854407296/t0(0) o251->MGC10.2.5.84@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1469502275 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4929.911272] Lustre: 14924:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4930.336132] Lustre: server umount lustre-MDT0000 complete [ 4930.512606] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4930.800484] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [ 4934.740718] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 4935.061369] Lustre: DEBUG MARKER: zpool list -H lustre-mdt1 >/dev/null 2>&1 || zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 4936.665977] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 4936.968834] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 4937.688972] LustreError: 16411:0:(obd_config.c:1397:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4938.311781] Lustre: lustre-MDD0000: changelog on [ 4938.490147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4938.796123] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4939.360896] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4939.663661] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4939.979075] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4940.289768] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 4940.618996] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4940.930151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4941.239537] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl1 [ 4941.403607] Lustre: lustre-MDD0000: changelog off [ 4941.560689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4941.877666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4942.190219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4945.250159] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4951.181879] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4951.370719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4951.747315] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4952.173484] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4952.700605] Lustre: DEBUG MARKER: dmesg [ 4953.551955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160b: Verify that very long rename doesn\'t crash in changelog ========================= 20:05:00 \(1469502300\) [ 4953.740574] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================= 20:05:00 (1469502300) [ 4953.967880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4954.301006] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4954.451746] Lustre: lustre-MDD0000: changelog on [ 4954.610510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4954.997700] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl2 [ 4955.158147] Lustre: lustre-MDD0000: changelog off [ 4955.364075] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4955.761068] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4956.297517] Lustre: DEBUG MARKER: dmesg [ 4956.920579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160c: verify that changelog log catch the truncate event ============================== 20:05:03 \(1469502303\) [ 4957.172507] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ============================== 20:05:03 (1469502303) [ 4957.380447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4957.700884] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4957.842124] Lustre: lustre-MDD0000: changelog on [ 4958.010569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4958.310862] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-TRUNC [ 4958.630534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4958.953863] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+TRUNC [ 4959.290852] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl3 [ 4959.443389] Lustre: lustre-MDD0000: changelog off [ 4959.643518] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4960.127090] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4960.607874] Lustre: DEBUG MARKER: dmesg [ 4961.173129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160d: verify that changelog log catch the migrate event =============================== 20:05:07 \(1469502307\) [ 4961.439283] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event =============================== 20:05:07 (1469502307) [ 4961.766263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_160d needs \>= 2 MDTs [ 4961.977857] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [ 4962.210889] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4962.651855] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4963.050924] Lustre: DEBUG MARKER: dmesg [ 4963.629760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161a: link ea sanity ================================================================== 20:05:10 \(1469502310\) [ 4963.823110] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ================================================================== 20:05:10 (1469502310) [ 4973.442808] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4973.915907] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4974.417400] Lustre: DEBUG MARKER: dmesg [ 4974.932682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161b: link ea sanity under remote directory =========================================== 20:05:21 \(1469502321\) [ 4975.144083] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory =========================================== 20:05:21 (1469502321) [ 4975.380883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_161b skipping remote directory test [ 4975.659658] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [ 4975.940283] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4976.400373] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4976.943155] Lustre: DEBUG MARKER: dmesg [ 4977.472580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 20:05:24 \(1469502324\) [ 4977.711298] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 20:05:24 (1469502324) [ 4977.919619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4978.220984] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4978.362388] Lustre: lustre-MDD0000: changelog on [ 4978.861160] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl4 [ 4979.014554] Lustre: lustre-MDD0000: changelog off [ 4979.229156] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4979.670523] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4980.128197] Lustre: DEBUG MARKER: dmesg [ 4980.790771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162a: path lookup sanity ============================================================== 20:05:27 \(1469502327\) [ 4981.041513] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ============================================================== 20:05:27 (1469502327) [ 4981.541517] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4982.003518] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4982.459432] Lustre: DEBUG MARKER: dmesg [ 4982.994672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162b: striped directory path lookup sanity ============================================ 20:05:29 \(1469502329\) [ 4983.231335] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ============================================ 20:05:29 (1469502329) [ 4983.538831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_162b needs \>= 2 MDTs [ 4983.781418] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [ 4984.008645] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4984.522312] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4984.966150] Lustre: DEBUG MARKER: dmesg [ 4985.591333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 20:05:32 \(1469502332\) [ 4985.793136] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 20:05:32 (1469502332) [ 4985.969904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4991.678202] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4992.121540] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4992.610133] Lustre: DEBUG MARKER: dmesg [ 4993.241947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 169: parallel read and truncate should not deadlock =================================== 20:05:40 \(1469502340\) [ 4993.428792] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock =================================== 20:05:40 (1469502340) [ 4993.683510] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating a 10 Mb file [ 4993.932548] Lustre: DEBUG MARKER: creating a 10 Mb file [ 4995.769659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark starting reads [ 4995.961179] Lustre: DEBUG MARKER: starting reads [ 4996.270591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating the file [ 4996.528612] Lustre: DEBUG MARKER: truncating the file [ 4996.790872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark killing dd [ 4997.019785] Lustre: DEBUG MARKER: killing dd [ 4997.257799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing the temporary file [ 4997.500525] Lustre: DEBUG MARKER: removing the temporary file [ 4997.777080] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4998.218285] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 4998.677865] Lustre: DEBUG MARKER: dmesg [ 4999.278564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 170: test lctl df to handle corrupted log ============================================= 20:05:45 \(1469502345\) [ 4999.503184] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ============================================= 20:05:45 (1469502345) [ 5001.786180] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5002.262530] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 5002.776179] Lustre: DEBUG MARKER: dmesg [ 5003.360126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit\(\) ============================== 20:05:50 \(1469502350\) [ 5003.622689] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ============================== 20:05:50 (1469502350) [ 5006.893720] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5007.322370] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 5007.781641] Lustre: DEBUG MARKER: dmesg [ 5008.305753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_180a skipping ALWAYS excluded test 180a \(base 180\) [ 5008.542577] Lustre: DEBUG MARKER: SKIP: sanity test_180a skipping ALWAYS excluded test 180a (base 180) [ 5008.764478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_180b skipping ALWAYS excluded test 180b \(base 180\) [ 5008.950764] Lustre: DEBUG MARKER: SKIP: sanity test_180b skipping ALWAYS excluded test 180b (base 180) [ 5009.170110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_180c skipping ALWAYS excluded test 180c \(base 180\) [ 5009.424162] Lustre: DEBUG MARKER: SKIP: sanity test_180c skipping ALWAYS excluded test 180c (base 180) [ 5009.739776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 181: Test open-unlinked dir =========================================================== 20:05:56 \(1469502356\) [ 5009.942364] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir =========================================================== 20:05:56 (1469502356) [ 5031.442141] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5031.868225] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); if [[ $? -eq 0 && $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc [ 5032.294285] Lustre: DEBUG MARKER: dmesg [ 5032.887400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 182: Test parallel modify metadata operations ========================================= 20:06:19 \(1469502379\) [ 5033.084558] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================= 20:06:19 (1469502379) [ 5068.051004] BUG: soft lockup - CPU#0 stuck for 22s! [osp-syn-0-0:16414] [ 5068.051004] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_generic crypto_null libcfs(OE) dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xprtrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic crct10dif_common ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ppdev virtio_balloon pcspkr ib_core ib_addr parport_pc i2c_piix4 parport zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) nfsd spl(OE) zlib_deflate nfs_acl auth_rpcgss lockd grace sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi cirrus syscopyarea sysfillrect sysimgblt [ 5068.069003] BUG: soft lockup - CPU#1 stuck for 23s! [mdt00_007:23527] [ 5068.051004] virtio_blk drm_kms_helper ttm ata_piix 8139too libata serio_raw drm virtio_pci virtio_ring [ 5068.069003] Modules linked in: [ 5068.069003] osp(OE) [ 5068.069003] mdd(OE) [ 5068.069003] lod(OE) [ 5068.069003] mdt(OE) [ 5068.069003] lfsck(OE) [ 5068.069003] mgs(OE) [ 5068.069003] mgc(OE) [ 5068.069003] osd_zfs(OE) [ 5068.069003] lquota(OE) [ 5068.069003] fid(OE) [ 5068.069003] fld(OE) [ 5068.069003] ksocklnd(OE) [ 5068.069003] ptlrpc(OE) [ 5068.069003] obdclass(OE) [ 5068.069003] lnet(OE) [ 5068.069003] sha512_generic [ 5068.069003] crypto_null [ 5068.069003] libcfs(OE) [ 5068.069003] dm_mod [ 5068.069003] rpcsec_gss_krb5 [ 5068.069003] nfsv4 [ 5068.069003] dns_resolver [ 5068.069003] nfs [ 5068.069003] fscache [ 5068.069003] xprtrdma [ 5068.069003] ib_isert [ 5068.069003] iscsi_target_mod [ 5068.069003] ib_iser [ 5068.069003] libiscsi [ 5068.069003] scsi_transport_iscsi [ 5068.069003] ib_srpt [ 5068.069003] target_core_mod [ 5068.069003] crc_t10dif [ 5068.069003] crct10dif_generic [ 5068.069003] crct10dif_common [ 5068.069003] ib_srp [ 5068.069003] scsi_transport_srp [ 5068.069003] scsi_tgt [ 5068.069003] ib_ipoib [ 5068.069003] rdma_ucm [ 5068.069003] ib_ucm [ 5068.069003] ib_uverbs [ 5068.069003] ib_umad [ 5068.069003] rdma_cm [ 5068.069003] ib_cm [ 5068.069003] iw_cm [ 5068.069003] ib_sa [ 5068.069003] ib_mad [ 5068.069003] ppdev [ 5068.069003] virtio_balloon [ 5068.069003] pcspkr [ 5068.069003] ib_core [ 5068.069003] ib_addr [ 5068.069003] parport_pc [ 5068.069003] i2c_piix4 [ 5068.069003] parport [ 5068.069003] zfs(POE) [ 5068.069003] zunicode(POE) [ 5068.069003] zavl(POE) [ 5068.069003] zcommon(POE) [ 5068.069003] znvpair(POE) [ 5068.069003] nfsd [ 5068.069003] spl(OE) [ 5068.069003] zlib_deflate [ 5068.069003] nfs_acl [ 5068.069003] auth_rpcgss [ 5068.069003] lockd [ 5068.069003] grace [ 5068.069003] sunrpc [ 5068.069003] ip_tables [ 5068.069003] ext4 [ 5068.069003] mbcache [ 5068.069003] jbd2 [ 5068.069003] ata_generic [ 5068.069003] pata_acpi [ 5068.069003] cirrus [ 5068.069003] syscopyarea [ 5068.069003] sysfillrect [ 5068.069003] sysimgblt [ 5068.069003] virtio_blk [ 5068.069003] drm_kms_helper [ 5068.069003] ttm [ 5068.069003] ata_piix [ 5068.069003] 8139too [ 5068.069003] libata [ 5068.069003] serio_raw [ 5068.069003] drm [ 5068.069003] virtio_pci [ 5068.069003] virtio_ring [ 5068.069003] virtio [ 5068.069003] 8139cp [ 5068.069003] i2c_core [ 5068.069003] mii [ 5068.069003] floppy [ 5068.069003] [last unloaded: llog_test] [ 5068.069003] [ 5068.069003] CPU: 1 PID: 23527 Comm: mdt00_007 Tainted: P OE ------------ 3.10.0-327.22.2.el7_lustre.x86_64 #1 [ 5068.069003] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [ 5068.069003] task: ffff88004f4e1700 ti: ffff880049c50000 task.ti: ffff880049c50000 [ 5068.069003] RIP: 0010:[] [ 5068.069003] [] __read_lock_failed+0xa/0x20 [ 5068.069003] RSP: 0018:ffff880049c53720 EFLAGS: 00000297 [ 5068.069003] RAX: ffff88007b91ecc0 RBX: 0000000000000000 RCX: ffff880049c53fd8 [ 5068.069003] RDX: 0000000000000000 RSI: ffff880047a1ca68 RDI: ffff880047a1cb20 [ 5068.069003] RBP: ffff880049c53720 R08: 00000000ffffffff R09: ffff88004e42fc00 [ 5068.069003] R10: ffff88004946a128 R11: 0000000000000000 R12: ffffffffa03f617c [ 5068.069003] R13: ffff880049c536b0 R14: ffff8800456e98f8 R15: ffff8800791ef228 [ 5068.069003] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [ 5068.069003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 5068.069003] CR2: 00007f08c2042298 CR3: 000000007b02d000 CR4: 00000000000006e0 [ 5068.069003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5068.069003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 5068.069003] Stack: [ 5068.069003] ffff880049c53730 [ 5068.069003] ffffffff8163d984 [ 5068.069003] ffff880049c53770 [ 5068.069003] ffffffffa0ecefa0 [ 5068.069003] [ 5068.069003] 00000000000001ab [ 5068.069003] 0000000000000001 [ 5068.069003] 000000000f114d24 [ 5068.069003] ffff880042724500 [ 5068.069003] [ 5068.069003] ffff880048bddd80 [ 5068.069003] ffff880048f96cb0 [ 5068.069003] ffff880049c53810 [ 5068.069003] ffffffffa0a5b25c [ 5068.069003] [ 5068.069003] Call Trace: [ 5068.069003] [ 5068.069003] [] _raw_read_lock+0x14/0x20 [ 5068.069003] [ 5068.069003] [] osd_attr_get+0x90/0x2d0 [osd_zfs] [ 5068.069003] [ 5068.069003] [] llog_osd_write_rec+0x1fc/0x1560 [obdclass] [ 5068.069003] [ 5068.069003] [] ? avl_find+0x5d/0xa0 [zavl] [ 5068.069003] [ 5068.069003] [] llog_write_rec+0xaa/0x280 [obdclass] [ 5068.069003] [ 5068.069003] [] llog_cat_add_rec+0x268/0x780 [obdclass] [ 5068.069003] [ 5068.069003] [] llog_add+0x7a/0x1a0 [obdclass] [ 5068.069003] [ 5068.069003] [] ? dnode_rele_and_unlock+0x5b/0x90 [zfs] [ 5068.069003] [ 5068.069003] [] osp_sync_add_rec.isra.16+0x270/0x9d0 [osp] [ 5068.069003] [ 5068.069003] [] osp_sync_add+0x48/0x50 [osp] [ 5068.069003] [ 5068.069003] [] osp_object_destroy+0xe4/0x220 [osp] [ 5068.069003] [ 5068.069003] [] lod_sub_object_destroy+0x1bb/0x450 [lod] [ 5068.069003] [ 5068.069003] [] lod_object_destroy+0x40b/0x690 [lod] [ 5068.069003] [ 5068.069003] [] ? osd_object_write_locked+0x3c/0x60 [osd_zfs] [ 5068.069003] [ 5068.069003] [] mdd_finish_unlink+0x123/0x410 [mdd] [ 5068.069003] [ 5068.069003] [] mdd_unlink+0xc78/0xd80 [mdd] [ 5068.069003] [ 5068.069003] [] mdt_reint_unlink+0xc24/0x11d0 [mdt] [ 5068.069003] [ 5068.069003] [] mdt_reint_rec+0x80/0x210 [mdt] [ 5068.069003] [ 5068.069003] [] mdt_reint_internal+0x5b2/0x9b0 [mdt] [ 5068.069003] [ 5068.069003] [] mdt_reint+0x67/0x140 [mdt] [ 5068.069003] [ 5068.069003] [] tgt_request_handle+0x915/0x1320 [ptlrpc] [ 5068.069003] [ 5068.069003] [] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] [ 5068.069003] [ 5068.069003] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 5068.069003] [ 5068.069003] [] ptlrpc_main+0xaa0/0x1de0 [ptlrpc] [ 5068.069003] [ 5068.069003] [] ? __switch_to+0xf8/0x4b0 [ 5068.069003] [ 5068.069003] [] ? ptlrpc_register_service+0xe40/0xe40 [ptlrpc] [ 5068.069003] [ 5068.069003] [] kthread+0xcf/0xe0 [ 5068.069003] [ 5068.069003] [] ? kthread_create_on_node+0x140/0x140 [ 5068.069003] [ 5068.069003] [] ret_from_fork+0x58/0x90 [ 5068.069003] [ 5068.069003] [] ? kthread_create_on_node+0x140/0x140 [ 5068.069003] Code: [ 5068.069003] 90 [ 5068.069003] 55 [ 5068.069003] 48 [ 5068.069003] 89 [ 5068.069003] e5 [ 5068.069003] f0 [ 5068.069003] ff [ 5068.069003] 07 [ 5068.069003] f3 [ 5068.069003] 90 [ 5068.069003] 83 [ 5068.069003] 3f [ 5068.069003] 01 [ 5068.069003] 75 [ 5068.069003] f9 [ 5068.069003] f0 [ 5068.069003] ff [ 5068.069003] 0f [ 5068.069003] 75 [ 5068.069003] f1 [ 5068.069003] 5d [ 5068.069003] c3 [ 5068.069003] 66 [ 5068.069003] 66 [ 5068.069003] 2e [ 5068.069003] 0f [ 5068.069003] 1f [ 5068.069003] 84 [ 5068.069003] 00 [ 5068.069003] 00 [ 5068.069003] 00 [ 5068.069003] 00 [ 5068.069003] 00 [ 5068.069003] 55 [ 5068.069003] 48 [ 5068.069003] 89 [ 5068.069003] e5 [ 5068.069003] f0 [ 5068.069003] 48 [ 5068.069003] ff [ 5068.069003] 07 [ 5068.069003] f3 [ 5068.069003] 90 [ 5068.069003] <48> [ 5068.069003] 83 [ 5068.069003] 3f [ 5068.069003] 01 [ 5068.069003] 78 [ 5068.069003] f8 [ 5068.069003] f0 [ 5068.069003] 48 [ 5068.069003] ff [ 5068.069003] 0f [ 5068.069003] 78 [ 5068.069003] ee [ 5068.069003] 5d [ 5068.069003] c3 [ 5068.069003] 90 [ 5068.069003] 90 [ 5068.069003] 90 [ 5068.069003] 90 [ 5068.069003] 90 [ 5068.069003] 90 [ 5068.069003] 90 [ 5068.069003] [ 5068.069003] Kernel panic - not syncing: softlockup: hung tasks [ 5068.069003] CPU: 1 PID: 23527 Comm: mdt00_007 Tainted: P OEL ------------ 3.10.0-327.22.2.el7_lustre.x86_64 #1 [ 5068.069003] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [ 5068.069003] ffffffff818731d4 [ 5068.069003] 000000000f114d24 [ 5068.069003] ffff88007fd03e18 [ 5068.069003] ffffffff81636324 [ 5068.069003] [ 5068.069003] ffff88007fd03e98 [ 5068.069003] ffffffff8162fb9a [ 5068.069003] 0000000000000008 [ 5068.069003] ffff88007fd03ea8 [ 5068.069003] [ 5068.069003] ffff88007fd03e48 [ 5068.069003] 000000000f114d24 [ 5068.069003] ffff88007fd03e67 [ 5068.069003] 0000000000000000 [ 5068.069003] [ 5068.069003] Call Trace: [ 5068.069003] [ 5068.069003] [ 5068.069003] [] dump_stack+0x19/0x1b [ 5068.069003] [ 5068.069003] [] panic+0xd8/0x1e7 [ 5068.069003] [ 5068.069003] [] watchdog_timer_fn+0x1b6/0x1c0 [ 5068.069003] [ 5068.069003] [] ? watchdog_enable+0xc0/0xc0 [ 5068.069003] [ 5068.069003] [] __hrtimer_run_queues+0xd2/0x260 [ 5068.069003] [ 5068.069003] [] hrtimer_interrupt+0xb0/0x1e0 [ 5068.069003] [ 5068.069003] [] ? call_softirq+0x1c/0x30 [ 5068.069003] [ 5068.069003] [] local_apic_timer_interrupt+0x37/0x60 [ 5068.069003] [ 5068.069003] [] smp_apic_timer_interrupt+0x3f/0x60 [ 5068.069003] [ 5068.069003] [] apic_timer_interrupt+0x6d/0x80 [ 5068.069003] [ 5068.069003] [ 5068.069003] [] ? __read_lock_failed+0xa/0x20 [ 5068.069003] [ 5068.069003] [] _raw_read_lock+0x14/0x20 [ 5068.069003] [ 5068.069003] [] osd_attr_get+0x90/0x2d0 [osd_zfs] [ 5068.069003] [ 5068.069003] [] llog_osd_write_rec+0x1fc/0x1560 [obdclass] [ 5068.069003] [ 5068.069003] [] ? avl_find+0x5d/0xa0 [zavl] [ 5068.069003] [ 5068.069003] [] llog_write_rec+0xaa/0x280 [obdclass] [ 5068.069003] [ 5068.069003] [] llog_cat_add_rec+0x268/0x780 [obdclass] [ 5068.069003] [ 5068.069003] [] llog_add+0x7a/0x1a0 [obdclass] [ 5068.069003] [ 5068.069003] [] ? dnode_rele_and_unlock+0x5b/0x90 [zfs] [ 5068.069003] [ 5068.069003] [] osp_sync_add_rec.isra.16+0x270/0x9d0 [osp] [ 5068.069003] [ 5068.069003] [] osp_sync_add+0x48/0x50 [osp] [ 5068.069003] [ 5068.069003] [] osp_object_destroy+0xe4/0x220 [osp] [ 5068.069003] [ 5068.069003] [] lod_sub_object_destroy+0x1bb/0x450 [lod] [ 5068.069003] [ 5068.069003] [] lod_object_destroy+0x40b/0x690 [lod] [ 5068.069003] [ 5068.069003] [] ? osd_object_write_locked+0x3c/0x60 [osd_zfs] [ 5068.069003] [ 5068.069003] [] mdd_finish_unlink+0x123/0x410 [mdd] [ 5068.069003] [ 5068.069003] [] mdd_unlink+0xc78/0xd80 [mdd] [ 5068.069003] [ 5068.069003] [] mdt_reint_unlink+0xc24/0x11d0 [mdt] [ 5068.069003] [ 5068.069003] [] mdt_reint_rec+0x80/0x210 [mdt] [ 5068.069003] [ 5068.069003] [] mdt_reint_internal+0x5b2/0x9b0 [mdt] [ 5068.069003] [ 5068.069003] [] mdt_reint+0x67/0x140 [mdt] [ 5068.069003] [ 5068.069003] [] tgt_request_handle+0x915/0x1320 [ptlrpc] [ 5068.069003] [ 5068.069003] [] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] [ 5068.069003] [ 5068.069003] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 5068.069003] [ 5068.069003] [] ptlrpc_main+0xaa0/0x1de0 [ptlrpc] [ 5068.069003] [ 5068.069003] [] ? __switch_to+0xf8/0x4b0 [ 5068.069003] [ 5068.069003] [] ? ptlrpc_register_service+0xe40/0xe40 [ptlrpc] [ 5068.069003] [ 5068.069003] [] kthread+0xcf/0xe0 [ 5068.069003] [ 5068.069003] [] ? kthread_create_on_node+0x140/0x140 [ 5068.069003] [ 5068.069003] [] ret_from_fork+0x58/0x90 [ 5068.069003] [ 5068.069003] [] ? kthread_create_on_node+0x140/0x140